Rendering multi-sensory experiences from object-based representations

Abstract

Some methods for controlling a set of controllable actuators may involve obtaining, by a control system, actuator data for the set of controllable actuators and receiving object-based sensory data and sensory object geometry metadata The object-based sensory data may include a set of one or more sensory objects and the sensory object geometry metadata may indicate a sensory object shape and a sensory object size of a corresponding sensory object of the set of one or more sensory objects. Some methods may involve rendering, by the control system, the object-based sensory data to produce one or more actuator control signals. The rendering may be based at least in part on the actuator data and the sensory object geometry metadata. Some methods may involve providing the one or more actuator control signals to one or more controllable actuators of the set of controllable actuators.

Description

D24100WO01RENDERING MULTI-SENSORY EXPERIENCES FROM OBJECT-BASED REPRESENTATIONSCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of priority from U. S. Provisional Application No.63 / 720,648, filed on 14 November 2024, U. S. Provisional Application No. 63 / 720,635, filed on 14 November 2024, U. S. Provisional Application No. 63 / 720,659, filed on 14 November 2024 and U. S. Provisional Application No. 63 / 784,898 filed on 7 April 2025, each of which is incorporated by reference herein in its entirety.TECHNICAL FIELD

[0002] The present disclosure relates to providing multi-sensory (MS) experiences, which are also referred to herein as multi-modal (MM) experiences, and is more specifically directed to aspects of an MS renderer.BACKGROUND

[0003] Unless otherwise indicated herein, the approaches described in this section are not prior art to the claims in this application and are not admitted as prior art by inclusion in this section.

[0004] Media content delivery has generally focused on audio and screen-based visual experiences. There has been limited delivery of multi-sensory content due to the bespoke nature of actuation. Luminaires, for example, are used extensively as an expression of art and function for concerts. However, each installation is designed specifically for a unique set of luminaires. Delivering a lighting design beyond the set of fixtures the system was designed for is generally not feasible. Other systems that attempt to deliver light experiences more broadly simply do so by extending the screen visuals algorithmically, but are not specifically authored. Haptics content is designed for a specific haptics apparatus. If another device, such as a game controller, mobile phone or even a different brand of haptics device is used, there has been no way to translate the creative intent of content to the different actuators.D24100WO01SUMMARY

[0005] At least some aspects of the present disclosure may be implemented via methods. In some instances, the methods may be implemented, at least in part, by a control system such as those disclosed herein. Some methods may involve controlling a set of one or more controllable actuators of a playback environment. Some methods may involve obtaining, by a control system, actuator data for the set of one or more controllable actuators. In some examples, the set of one or more controllable actuators may include one or more light fixtures, one or more haptic devices, one or more air flow control devices, or combinations thereof.

[0006] Some methods may involve receiving, by the control system, object-based sensory data and sensory object geometry metadata. In some examples, the object-based sensory data may include a set of one or more sensory objects and the sensory object geometry metadata may indicate a sensory object shape and a sensory object size of a corresponding sensory object of the set of one or more sensory objects. In some instances, the object-based sensory data may include other types of sensory object metadata.

[0007] Some methods may involve rendering, by a sensory data renderer implemented via the control system, the object-based sensory data to produce one or more actuator control signals. According to some examples, the rendering may be based at least in part on the actuator data and the sensory object geometry metadata. Some methods may involve providing, by the control system, the one or more actuator control signals to one or more controllable actuators of the set of controllable actuators.

[0008] In some examples, the sensory object geometry metadata may include feathering information. The feathering information may define a feathering region in which an actuator activation value is between zero and one, wherein one is a maximum value. The feathering information may include a feathering function to be applied in one dimension, two dimensions or three dimensions. The feathering information may be based, at least in part, on a normal to one or more sensory object surfaces, on a normal to one or more surfaces that define a feathering volume, or combinations thereof. In some examples, the sensory object geometry metadata may include a two-dimensional shape or a three-dimensional shape within which the actuator activation value is one. The two-dimensional shape or the three-dimensional shape may reside within the feathering region.D24100WO01

[0009] According to some examples, the rendering may be based, at least in part, on a precomputed data structure. In some examples, the data structure may be an acceleration data structure.

[0010] In some examples, the set of one or more controllable actuators may include a set of one or more light fixtures, and the object-based sensory data may include light object data and light object geometry metadata. In some such examples, the sensory data Tenderer may be, or may include, a lightscape renderer configured to produce control signals for the set of one or more light fixtures based at least in part on the light object geometry metadata and the actuator data.

[0011] According to some examples, the actuator data may include extended light activation volume data for one or more extended light fixtures. In some examples, the extended light activation volume data may indicate one or more luminance volumes of the playback environment. Each of the one or more luminance volumes may be a volume of the playback environment in which a corresponding extended light fixture is estimated to cause luminance that is perceivable by a human viewer. According to some examples, the one or more luminance volumes may include one or more volumes illuminated by direct light from the extended light fixture, one or more volumes illuminated by reflected light from the extended light fixture, or combinations thereof.

[0012] In some examples, the extended light activation volume data may include one or more rotation parameters corresponding to rotation of the corresponding extended light fixture. According to some examples, the rendering may involve determining an activation of an extended light fixture by computing an inner product of extended light activation volume data and light object geometry metadata in an activation space.

[0013] According to some examples, the light object geometry metadata may include a set of light object coordinates. In some such examples, the rendering may involve determining an activation of an extended light fixture by projecting at least one object coordinate of the set of light object coordinates onto an extended light activation volume indicated by the extended light activation volume data. In some such examples, the rendering may involve determining the activation of the extended light fixture by projecting a set of test light object coordinates of the set of light object coordinates onto the extended light activation volume and evaluating activations of a corresponding set of projected test light object coordinates.

[0014] Some or all of the operations, functions and / or methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more computer-readable non-transitory media. Such non-transitory media may include oneD24100WO01or more memory devices such as those described herein, including but not limited to one or more random access memory (RAM) devices, read-only memory (ROM) devices, etc. Accordingly, some innovative aspects of the subject matter described in this disclosure can be implemented in one or more computer-readable non-transitory media having software stored thereon.

[0015] At least some aspects of the present disclosure may be implemented via apparatus. For example, one or more devices may be capable of performing, at least in part, the methods disclosed herein. In some implementations, an apparatus may include an interface system and a control system. The control system may include one or more general purpose single- or multi-chip processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic, discrete hardware components, or combinations thereof. The control system may be configured to perform some or all of the disclosed methods.

[0016] Details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages will become apparent from the description, the drawings, and the claims. Note that the relative dimensions of the following figures may not be drawn to scale.BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Disclosed embodiments now be described, by way of example only, with reference to the accompanying drawings.

[0018] Figure 1 A is a block diagram that shows examples of components of an apparatus capable of implementing various aspects of this disclosure.

[0019] Figure IB shows example elements of an endpoint.

[0020] Figure 2 shows examples of actuator elements.

[0021] Figure 3 shows example elements of a system for the creation and playback of multi-sensory (MS) experiences.

[0022] Figure 4 shows example elements of a multi-sensory (MS) renderer.

[0023] Figure 5 shows example elements of another system for the creation and playback of MS experiences.

[0024] Figure 6A shows an example light map for a table lamp.

[0025] Figure 6B shows an example of an egocentric light map.D24100WO01

[0026] Figure 7 shows elements of a lightscape renderer according to some examples.

[0027] Figure 8A shows an example of a playback environment within which are examples of a parameterizable sensory object and a light fixture.

[0028] Figure 8B shows an example of the playback environment of Figure 8A in which the parameterizable sensory object includes feathering.

[0029] Figures 9A, 9B and 9C show examples of the playback environment of Figure 8A in which a sensory object is defined by a set of vertices.

[0030] Figure 10 shows examples of extended light fixtures in a playback environment.

[0031] Figures 11 A, 1 IB and 11C show examples of projecting light object coordinates onto extended light activation volumes.

[0032] Figure 12 is a flow diagram that outlines one example of a method that may be performed by an apparatus or system such as those disclosed herein.

[0033] Figure 13 shows an example of an image asset that is associated with a sensory object.

[0034] Figures 14 and 15 show examples of the image asset of Figure 13 being rendered in different playback environments.

[0035] Figures 16A and 17A show examples of light objects in a reference environment.

[0036] Figures 16B and 17B show examples of the light objects of 16A and 17A projected into a playback environment.

[0037] Figure 18A shows another example of a reference environment with reference to which content can be created.

[0038] Figure 18B shows an example of an actuator domain for content creation in the reference environment of Figure 18 A.

[0039] Figure 19A shows another example of a reference environment with reference to which content can be created.

[0040] Figures 19B and 19C show examples of actuator domains for content creation in the reference environment of Figure 19A.

[0041] Figure 20 is a flow diagram that outlines one example of a method that may be performed by an apparatus or system such as those disclosed herein.

[0042] Figure 21 shows example elements of a system for simultaneously rendering a plurality of multi-modal bitstreams.

[0043] Figures 22A and 22B show example states of a state machine that may be implemented by the system of Figure 21 according to some implementations.D24100WO01

[0044] Figure 23 is a flow diagram that outlines one example of a method that may be performed by an apparatus or system such as those disclosed herein.DETAILED DESCRIPTION

[0045] Described herein are techniques related to providing multi-sensory media content. In the following description, for purposes of explanation, numerous examples and specific details are set forth in order to provide a thorough understanding of the present disclosure. It will be evident, however, to one skilled in the art that the present disclosure as defined by the claims may include some or all of the features in these examples alone or in combination with other features described below, and may further include modifications and equivalents of the features and concepts described herein.

[0046] In the following description, various methods, processes and procedures are detailed. Although particular steps may be described in a certain order, such order is mainly for convenience and clarity. A particular step may be repeated more than once, may occur before or after other steps (even if those steps are otherwise described in another order), and may occur in parallel with other steps. A second step is required to follow a first step only when the first step must be completed before the second step is begun. Such a situation will be specifically pointed out when not clear from the context.

[0047] In this document, the terms “and”, “or” and “and / or” are used. Such terms are to be read as having an inclusive meaning. For example, “A and B” may mean at least the following: “both A and B”, “at least both A and B”. As another example, “A or B” may mean at least the following: “at least A”, “at least B”, “both A and B”, “at least both A and B”. As another example, “A and / or B” may mean at least the following: “A and B”, “A or B”. When an exclusive-or is intended, such will be specifically noted (e.g., “either A or B”, “at most one of A and B”).

[0048] This document describes various processing functions that are associated with structures such as blocks, elements, components, circuits, etc. In general, these structures may be implemented by one or more processors controlled by one or more computer programs.D24100WO01

[0049] As noted above, media content delivery has generally been focused on audio and video experiences. There has been limited delivery of multi-sensory (MS) content due to the customized nature of actuation.

[0050] This application describes methods for extending the creative palette for content creators, allowing spatial, MS experiences to be created and delivered at scale. Some such methods involve the introduction of new layers of abstraction, in order to allow authored MS experiences to be delivered to different endpoints, with different types of fixtures or actuators. As used herein, the term “endpoint” is synonymous with “playback environment” or simply “environment,” meaning an environment that includes one or more actuators that may be used to provide an MS experience. Such endpoints may include a room, such as the living room of a home, a car, a cinema, a night club or other venue, etc. Some disclosed methods involve the creation, delivery and / or rendering of object-based sensory data, which may include sensory objects and corresponding sensory metadata. This abstraction allows creative intent to be implemented in an object-based format that does not require prior knowledge of the specific controller actuation, thereby enabling greater flexibility and scalability of fixtures and actuators across endpoints. An MS experience provided via object-based sensory data may be referred to herein as a “flexibly-scaled MS experience.”

[0051] Figure 1A is a block diagram that shows examples of components of an apparatus capable of implementing various aspects of this disclosure. As with other figures provided herein, the types and numbers of elements shown in Figure 1A are merely provided by way of example. Other implementations may include more, fewer and / or different types and numbers of elements. According to some examples, the apparatus 101 may be, or may include, a device that is configured for performing at least some of the methods disclosed herein, such as a smart audio device, a laptop computer, a cellular telephone, a tablet device, a smart home hub, etc. In some such implementations the apparatus 101 may be, or may include, a server that is configured for performing at least some of the methods disclosed herein.

[0052] In this example, the apparatus 101 includes at least an interface system 105 and a control system 110. In some implementations, the control system 110 may be configured for performing, at least in part, the methods disclosed herein. In some examples, the control system 110 may be configured for obtaining, via the interface system 105, actuator data for a set of controllable actuators. The set of controllable actuators may, for example, be specificD24100WO01to a particular playback environment. According to some examples, the control system 110 may be configured for obtaining, via the interface system 105, object-based sensory data including a set of sensory objects. In some examples, the object-based sensory data may include object-based sensory metadata corresponding to some or all of the sensory objects. The encoded object-based sensory data may correspond to sensory effects such as lighting, haptics, airflow, one or more positional actuators, or combinations thereof, to be provided by a plurality of sensory actuators in an environment.

[0053] According to some examples, the control system 110 may be configured for implementing a multi-sensory (MS) renderer. Accordingly, in some examples, the control system 110 may be configured for rendering the object-based sensory data to produce actuator control signals, wherein the rendering is based at least in part on the actuator data. The MS renderer also may be referred to herein as a “sensory Tenderer,” because in some instances the MS tenderer may be rendering only one type of MS data, such as object-based lighting data. According to some examples, the control system 110 may be configured for sending the actuator control signals to one or more controllable actuators of the set of controllable actuators.

[0054] According to some examples, the object-based sensory metadata may include sensory spatial metadata indicating at least a spatial position for rendering the object-based sensory metadata within the environment, an area for rendering the object-based sensory metadata within the environment, or combinations thereof. In some implementations, the object-based sensory metadata does not correspond to any particular sensory actuator in the environment. In some examples, the object-based sensory metadata may include abstracted sensory reproduction information allowing the sensory renderer to reproduce authored sensory effects, which also may be referred to herein as intended sensory effects, via various sensory actuator types, via various numbers of sensory actuators and from various sensory actuator positions in the environment.

[0055] In some examples, the control system 110 may be configured for obtaining, via the interface system 105, local context information. The local context information may include local time of day information, local weather information, local human behavior information, local user input, information regarding one or more viewer preferences, information regarding presence or absence of one or more viewers, ambient light information, viewing environment information, local viewer location information, local device usage information,D24100WO01local viewer activity information, or combinations thereof. In some such examples, the rendering process may be based, at least in part, on the local context information.

[0056] In some examples, the content bitstream also may include encoded audio objects synchronized with the encoded object -based sensory metadata. The audio objects may include audio signals and corresponding audio object metadata. According to some examples, the audio objects may include audio signals and corresponding audio object metadata. The audio object metadata may include at least audio object spatial metadata indicating an audio object spatial position for rendering the audio signals within the environment. In some examples, the MS renderer also may be configured for rendering the audio objects.

[0057] The interface system 105 may include one or more network interfaces and / or one or more external device interfaces (such as one or more universal serial bus (USB) interfaces). According to some implementations, the interface system 105 may include one or more wireless interfaces. The interface system 105 may include one or more devices for implementing a user interface, such as one or more microphones, one or more speakers, a display system, a touch sensor system and / or a gesture sensor system. In some examples, the interface system 105 may include one or more interfaces between the control system 110 and a memory system, such as the optional memory system 115 shown in Figure 1A.However, the control system 110 may include a memory system in some instances.

[0058] The control system 110 may, for example, include a general purpose single- or multichip processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, and / or discrete hardware components.

[0059] In some implementations, the control system 110 may reside in more than one device. For example, a portion of the control system 110 may reside in a device within an environment (such as a laptop computer, a tablet computer, a smart audio device, etc.) and another portion of the control system 110 may reside in a device that is outside the environment, such as a server. In other examples, a portion of the control system 110 may reside in a device within an environment and another portion of the control system 110 may reside in one or more other devices of the environment.D24100WO01

[0060] Some or all of the methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include memory devices such as those described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, etc. The one or more non-transitory media may, for example, reside in the optional memory system 115 shown in Figure 1A and / or in the control system 110. Accordingly, various innovative aspects of the subject matter described in this disclosure can be implemented in one or more non-transitory media having software stored thereon. The software may, for example, include instructions for controlling at least one device to process audio data. The software may, for example, be executable by one or more components of a control system such as the control system 110 of Figure 1 A.

[0061] In some examples, the apparatus 101 may include the optional microphone system 120 shown in Figure 1A. The optional microphone system 120 may include one or more microphones. In some implementations, one or more of the microphones may be part of, or associated with, another device, such as a speaker of the speaker system, a smart audio device, etc.

[0062] According to some implementations, the apparatus 101 may include the optional actuator system 125 shown in Figure 1A. The optional actuator system 125 may include one or more loudspeakers, one or more haptic devices, one or more light fixtures, also referred to herein as luminaires, one or more fans or other air-moving devices, one or more display devices, including but not limited to one or more televisions, one or more positional actuators, one or more other types of devices for providing a MS experience, or combinations thereof. The term “light fixture” as used herein refers generally to any actuator that is configured to provide light. The term “light fixture” encompasses various types of light sources, including individual light sources such as light bulbs, groups of light sources such as light strips, light panels such as light-emitting diode (LED) panels, projectors, display devices such as television (TV) screens, etc. A “light fixture” may be moveable, and therefore the word “fixture" in this context does not mean that a light fixture is necessarily in a fixed position in space. The term “positional actuators” as used herein refers generally to devices that are configured to change a position or orientation of a person or object, such as motion simulator seats. Loudspeakers may sometimes be referred to herein as “speakers.” In some implementations, the optional actuator system 125 may include a display system including one or more displays, such as one or more light-emittingD24100WO01diode (LED) displays, one or more organic light-emitting diode (OLED) displays, etc. In some examples wherein the apparatus 101 includes a display system, the optional sensor system 130 may include a touch sensor system and / or a gesture sensor system proximate one or more displays of the display system. According to some such implementations, the control system 110 may be configured for controlling the display system to present a graphical user interface (GUI), such as a GUI related to implementing one of the methods disclosed herein.

[0063] In some implementations, the apparatus 101 may include the optional sensor system 130 shown in Figure 1 A. The optional sensor system 130 may include a touch sensor system, a gesture sensor system, one or more cameras, etc.

[0064] This application describes methods for rendering and delivering a flexibly scaled multi-sensory (MS) immersive experience (MS IE) to different playback environments, which also may be referred to herein as endpoints. Such endpoints may include a room, such as the living room of a home, a car, a cinema, a night club or other venue, an AR / VR headset, a PC, a mobile device, etc.

[0065] Figure IB shows example elements of an endpoint. In this example, the endpoint is a living room 1001 containing multiple actuators 008, some furniture 1010 and a person 1000 — also referred to herein as a user — who will consume a flexibly-scaled MS experience. Actuators 008 are devices capable of altering the environment 1001 that the user 1000 is in. Actuators 008 may include one or more haptic devices, one or more light fixtures, also referred to herein as luminaires, one or more fans or other air-moving devices, one or more display devices, including but not limited to one or more televisions, one or more positional actuators, one or more other types of devices for providing a MS experience, or combinations thereof.

[0066] The number of actuators 008, the arrangement of actuators 008 and the capabilities of actuators 008 in the space 1001 may vary significantly between different endpoint types. For example, the number, arrangement and capabilities of actuators 008 in a car will generally be different from the number, arrangement and capabilities of actuators 008 in a living room, a night club, etc. In many implementations, the number, arrangement and / or capabilities of actuators 008 may vary significantly between different instances of the same type, e.g., between a small living room with 2 actuators 008 and a large living room with 16D24100WO01actuators 008. The present disclosure describes various methods for creating and delivering flexibly-scaled MSIEs to these non-homogenous endpoints.

[0067] Figure 2 shows examples of actuator elements. In this example, the actuator is a luminaire 1100, which includes a network module 1101, a control module 1102 and a light emitter 1103. According to this example, the light emitter 1103 includes one or more lightemitting devices, such as light-emitting diodes, which are configured to emit light into an environment in which the luminaire 1100 resides. In this example, the network module 1101 is configured to provide network connectivity to one or more other devices in the space, such as a device that sends commands to control the emission of light by the luminaire 1100. According to this example, the network module 1101 is an instance of the interface system 105 of Figure 1 A. In this example, the control module 1102 is configured to receive signals via the network module 1101 and to control the light emitter 1103 accordingly. According to this example, the control module 1102 is an instance of the control system 110 of Figure 1A.

[0068] Other examples of actuators also may include a network module 1101 and a control module 1102, but may include other types of actuating elements. Some such actuators may include one or more haptic devices, one or more fans or other air-moving devices, one or more positional actuators, one or more loudspeakers, one or more display devices, etc.

[0069] Figure 3 shows example elements of a system for the creation and playback of multi-sensory (MS) experiences. As noted elsewhere, the terms “multi-sensory” and “multimodal” are used synonymously herein. As with other figures provided herein, the types and numbers of elements shown in Figure 3 are merely provided by way of example. Other implementations may include more, fewer and / or different types and numbers of elements. According to some examples, system 300 may be, or may include, one or more devices configured for performing at least some of the methods disclosed herein. In some examples, system 300 may include one or more instances of the control system 110 of Figure 1 A that are configured for performing at least some of the methods disclosed herein.

[0070] According to the examples in the present disclosure, creating and providing an object-based MS Immersive Experience (MSIE) approach involves the application of a suite of technologies for creation, delivery and rendering of object-based sensory data, which may include sensory objects and corresponding sensory metadata, to the actuators 008. Some examples are described in the following paragraphs.D24100WO01

[0071] Object-Based Representation: In various disclosed implementations, multi-sensory (MS) effects are represented using what may be referred to herein as multi-sensory (MS) objects, or simply as “sensory objects.” According to some such implementations, properties such as layer-type and priority may be assigned to and associated with attached to each sensory object, enabling content creators’ intent to be represented in the rendered experiences. Detailed examples of sensory object properties are described below.

[0072] In this example, system 300 includes a content creation tool 000 that is configured for designing multi-sensory (MS) immersive content and for outputting object-based sensory data 005, either separately or in conjunction with corresponding audio data Oil and / or video data 012, depending on the particular implementation. The object-based sensory data 005 may include time stamp information, as well as information indicating the type of sensory object, the sensory object properties, etc. In this example, the object-based sensory data 005 is not “channel-based” data that corresponds to one or more particular sensory actuators in a playback environment, but instead is generalized for a wide range of playback environments with a wide range of actuator types, numbers of actuators, etc. In some examples, the object-based sensory data 005 may include object-based light data, object-based haptic data, object-based air flow data, or object-based positional actuator data, object-based olfactory data, object-based smoke data, object based data for one or more other types of sensor effects, or combinations thereof. According to some examples, the object-based sensory data 005 may include sensory objects and corresponding sensory metadata. For example, if the object-based sensory data 005 includes object-based light data, the object-based light data may include light object position metadata, light object color metadata, light object size metadata, light object intensity metadata, light object shape metadata, light object diffusion metadata, light object gradient metadata, light object priority metadata, light object layer metadata, or combinations thereof. In some examples, the object-based sensory data 005 may include time data, such as time stamp information. Although the content creation tool 000 is shown providing a stream of object-based sensory data 005 to the experience player 002 in this example, in alternative examples the content creation tool 000 may produce object-based sensory data 005 that is stored for subsequent use. Examples of graphical user interfaces for a light-object-based content creation tool are described below.Examples of MS Object Properties

[0073] Following is a non-exhaustive list of possible properties of MS objects:D24100WO01• Priority;• Layer;• Mixing Mode;• Persistence;• Effect; and• Spatial Panning Law.

[0074] EffectAs used herein, the “effect” of a MS object is a synonym for the type of MS object. An “effect” is, or indicates, the sensory effect that the MS object is providing. If an MS object is a light object, its effect will involve providing direct or indirect light. If an MS object is a haptic object, its effect will involve providing some type of haptic feedback. If an MS object is an air flow object, its effect will involve providing some type of air flow. As described in more detail below, some examples involve other “effect” categories.D24100WO01

[0075] PersistenceSome MS objects may contain a persistence property in their metadata. For example, as an moveable MS object moves around in a scene, the moveable MS object may persist for some period of time at locations that the moveable MS object passes through. That period of time may be indicated by persistence metadata. In some implementations, the MS renderer is responsible for constructing and maintaining the persistence state.

[0076] LayersAccording to some examples, individual MS objects may be assigned to “layers,” in which MS objects are grouped together according to one or more shared characteristics. For example, layers may group MS objects together according to their intended effect or type, which may include but are not limited to the following:Mood / AmbienceInformationalPunctuational / AttentionAlternatively, or additionally, in some examples, layers may be used to group MS objects together according to shared properties, which may include but are not limited to the following:- ColorIntensitySizeShapePositionRegion in space

[0077] PriorityIn some examples, MS objects may have a priority property that enables the Tenderer to detemrine which object(s) should take priority in an environment in which MS objects are contending for limited actuators. For example, if multiple light objects overlap with a single light fixture at a time during which all of the light objects are scheduled to be rendered, a Tenderer may refer to the priority of each light object in order to determine which light object(s) will be rendered. In some examples, priority may be defined between layers or within layers. According to some examples, priority may be linked to specific properties such as intensity. In some examples, priority may be defined temporally: for example, theD24100WO01most recent MS object to be rendered may take precedence over MS objects that have been rendered earlier. According to some examples, priority may be used to specify MS objects or layers that should be rendered regardless of the limitations of a particular actuator system in a playback environment.

[0078] Spatial Panning LawsSpatial panning laws may define a MS object’s movement across a space, how a MS object affects actuators as it moves between them, etc.

[0079] Mixing ModeThe mixing mode may specify how multiple objects are multiplexed onto a single actuator. In some examples, mixing modes may include one or more of the following:Max mode: select the MS object which activates an actuator the most;Mix mode: mix in some or all the objects according to a rule set, for example by summing activation levels, taking the average of activation levels, mixing color according to activation level or priority level, etc.;MaxNmix: mix in the top N MS objects (by activation level), according to a rule set.

[0080] According to some examples, more general metadata for an entire multi-sensory content file, instead of (or in addition to) per-object metadata may be defined. For example, MS content files may include metadata such as trim passes or mastering environment.Trim Controls

[0081] What are referred to in the context of Dolby Vision™ as “trim controls” may act as guidance on how to modulate the default rendering algorithm for specific environments or conditions at the endpoint. Trim controls may specify ranges and / or default values for various properties, including saturation, tone detail, gamma, etc. For example, there may be automotive trim controls, which provide specific defaults and / or rule sets for rendering in automotive environments, for example guidance that includes only objects of a certain priority or layer. Other examples may provide trim controls for environments with limited, complex or sparse multisensory actuators.D24100WO01

[0082] Mastering EnvironmentA single piece of multisensory content may include metadata on the properties of the mastering environment such as room size, reflectivity and ambient bias lighting level. The specific properties may differ depending on the desired endpoint actuators. Mastering environment information can aid in providing reference points for rendering in a playback environment.

[0083] MS Object Renderer: Various disclosed implementations provide a renderer that is configured render MS effects to actuators in a playback environment. According to this example, system 300 includes a MS renderer 001 that is configured to render object-based sensory data 005 to actuator control signals 310, based at least in part on environment and actuator data 004. In this example, the MS renderer 001 is configured to output the actuator control signals 310 to MS controllers 003, which are configured to control the actuators 008. In some examples, the MS renderer 001 may be configured to receive light objects and object-based lighting metadata indicating an intended lighting environment, as well as lighting information regarding a local lighting environment. The lighting information is one general type of environment and actuator data 004, and may include one or more characteristics of one or more controllable light sources in the local lighting environment. In some examples, the MS renderer 001 may be configured to determine a drive level for each of the one or more controllable light sources that approximates the intended lighting environment. According to some examples, the MS renderer 001 (or one of the MS controllers 003) may be configured to output the drive level to at least one of the controllable light sources. Some alternative examples may include a separate renderer for each type of actuator 008, such as one renderer for light fixtures, another renderer for haptic devices, another renderer for air flow devices, etc. In other implementations, a single renderer may be configured as a MS renderer and as an audio renderer and / or as a video renderer. In some implementations, the MS renderer 001 may be configured to adapt to changing conditions. Some examples of MS renderer 001 implementations are described in more detail below.

[0084] The environment and actuator data 004 may include what are referred to herein as “room descriptors” that describe actuator locations (e.g., according to an x,y,z coordinate system or a spherical coordinate system). In some examples, the environment and actuator data 004 may indicate actuator orientation and / or placement properties (e.g., directional andD24100WO01north-facing, omnidirectional, occlusion information, etc.). According to some examples, the environment and actuator data 004 may indicate actuator orientation and / or placement properties according to a 3x3 matrix, in which three elements (for example, the elements of the first row) represent spatial position (x,y,z), three other elements (for example, the elements of the second row) represent orientation (roll, pitch, yaw), and three other elements (for example, the elements of the third row) indicate a scale or size (sx, sy, sz). In some examples, the environment and actuator data 004 may include device descriptors that describe the actuator properties relevant to the MS renderer 001, such as intensity range and color gamut of a light fixture, the air flow speed range and direction(s) for an air-moving device, etc.

[0085] In this example, system 300 includes an experience player 002 that is configured to receive object-based sensory data 005’, audio data 011’ and video data 012’, and to provide object-based sensory data 005 to the MS renderer 001, to provide audio data 011 to the audio Tenderer 006 and to provide the video data 012 to the video Tenderer 007. In this example, the reference numbers for the object-based sensory data 005’, audio data 011’ and video data 012’ received by the experience player 002 include primes (‘), in order to suggest that the data may in some instances be encoded. Likewise, the object-based sensory data 005, audio data 011 and video data 012 output by the experience player 002 do not include primes, in order to suggest that the data may in some instances have been decoded by the experience player 002. According to some examples, the experience player 002 may be a media player, a game engine or personal computer or mobile device, or a component integrated in an television, DVD player, sound bar, set top box, or a service provider media device such as a Chromecast, Apple TV device, or Amazon Fire TV. In some examples, the experience player 002 may be configured to receive encoded object-based sensory data 005’ along with encoded audio data Oi l’ and / or encoded video data 012’. In some such examples, the encoded object-based sensory data 005’ may be received as part of the same bitstream with the encoded audio data Oil’ and / or the encoded video data 012’. Some examples are described in more detail below. According to some examples, the experience player 002 may be configured to extract the object-based sensory data 005’ from the content bitstream and to provide decoded object-based sensory data 005 to the MS renderer 001, to provide decoded audio data 011 to the audio Tenderer 006 and to provide decoded video data 012 to the video Tenderer 007. In some examples, time stamp information in the object-based sensory data 005' may be used — for example, by the experience player 102, the MSD24100WO01Tenderer 001, the audio Tenderer 106, the video Tenderer 107, or all of them — to synchronize effects relating to the object-based sensory data 005’ with the audio data 111’ and / or the video data 112’, which may also include time stamp information.

[0086] According to this example, system 300 includes MS controllers 003 that are configured to communicate with a variety of actuator types using application program interfaces (APIs) or one or more similar interfaces. Generally speaking, each actuator will require a specific type of control signal to produce the desired output from the Tenderer. According to this example, the MS controllers 003 are configured to map outputs from the MS tenderer 001 to control signals for each actuator. For example, a Philips Hue™ light bulb receives control information in a particular format to turn the light on, with a particular saturation, brightness and hue, and a digital representation of the desired drive level. In some alternative examples, the MS tenderer 001 also may be configured to implement some or all of the MS controllers 003. For example, the MS tenderer 001 also may be configured to implement one or more lighting-based APIs but not haptic-based APIs, or vice versa.

[0087] In some examples, room descriptors also may describe the size and orientation of the playback environment itself, to establish a relative or absolute coordinate system to which all objects are positioned. For example, in a living room a display screen may be regarded as the front, in some instances the front and center, and the floor and ceiling may be regarded as the vertical bounds. In some such examples, the room descriptors also may also indicate bounds corresponding with the left, right, front, and rear, walls relative to the front position. According to some examples, the room descriptor also may be provided in terms of a matrix, such as a 3x3 matrix. This room descriptor information is useful in describing the physical dimensions of the playback environment, for example in physical units of distance such as meters. In some such examples, sensory object locations, sensory object sizes, and sensory object orientations may be described in units that are relative to the room size, for example in a range from - 1 to 1. Room descriptors may also describe a preferred viewing position, in some instances according to a matrix.

[0088] The types, numbers and arrangements of the actuators 008 will generally vary according to the particular implementation. In some examples, actuators 008 may include lights and / or light strips (also referred to herein as “luminaires”), vibrational motors, air flow generators, positional actuators, or combinations thereof.D24100WO01

[0089] Similarly, the types, numbers and arrangements of the loudspeakers 009 and the display devices 010 will generally vary according to the particular implementation. In the examples shown in Figure 3, audio data 011 and video data 012 are rendered by the audio Tenderer 006 and the video Tenderer 007 to the loudspeakers 009 and display devices 010, respectively.

[0090] As noted above, according to some implementations the system 300 may include one or more instances of the control system 110 of Figure 1 A that are configured for performing at least some of the methods disclosed herein. In some such examples, one instance of the control system 110 may implement the content creation tool 000 and another instance of the control system 110 may implement the experience player 002. In some examples, one instance of the control system 110 may implement the audio Tenderer 006, the video Tenderer 007, the multi-sensory Tenderer 001, or combinations thereof. According to some examples, an instance of the control system 110 that is configured to implement the experience player 002 may also be configured to implement the audio Tenderer 006, the video Tenderer 007, the multi-sensory Tenderer 001, or combinations thereof.

[0091] Figure 4 shows example elements of a multi-sensory (MS) renderer. As with other figures provided herein, the types and numbers of elements shown in Figure 4 are merely provided by way of example. Other implementations may include more, fewer and / or different types and numbers of elements. According to this example, the MS renderer 001 is an instance of the MS renderer 001 that is described with reference to Figure 3. In some examples, the MS renderer 001 may be implemented by one or more instances of the control system 110 of Figure 1A.

[0092] According to this example, Figure 4 includes the following elements:• 004: Environment and actuator data, which may be as described with reference to Figure 3;• 005: Object-based sensory data 005, which may be as described with reference to Figure 3;• 423 an actuator map (AM) that indicates the locations of at least the controllable actuators 008 in a particular playback environment;• 450 a projection module configured to project the MS objects of the object-based sensory data 005 based, at least in part, on the AM 423. The MS objects may also be referred to herein as “sensory objects,’’ because in some instances only one type ofD24100WO01sensory object — such as only haptic objects or only light objects — may be present in the object-based sensory data 005. In this example, the projection module 450 is configured to project the MS objects based, at least in part, on sensory object metadata, which may include at least sensory object location metadata and sensory object size metadata;• 440 an actuator activation matrix (AAM) that is output by the projection module 450 according to this example. The AAM may, for example, indicate sensory objects, if any, that are currently encompassing a volume within the playback environment corresponding to one or more corresponding actuators. For example, the AAM may indicate whether the light object location metadata and light object size metadata of a light object indicate that a particular light fixture is within a volume of the playback environment corresponding to the location and size of the light object;• 451 a mixing module configured to convert the AAM 440 into actuator control signals, based at least in part on the environment and actuator data and the Tenderer configuration data;• 452 optional Tenderer configuration data, which may include information regarding one or more settings for the MS renderer 001 such as settings to indicate the desired dynamism, mode, etc. In some examples, the Tenderer configuration data 452 may be changed automatically using context-aware systems such as those described in more detail below; and• 310: actuator control signals, which may be as described with reference to Figure 3.In some examples, the actuator control signals 310 may be sent to individual actuators 008, whereas in other examples the actuator control signals 310 may be sent to MS controllers 003, which may be configured to send appropriate control signals to various types of actuators 008.

[0093] According to some examples, the AAM 440 is a matrix describing “how much” a sensory object projects itself onto each actuator according to the actuator map 423. In some examples, the AAM 440 may be a real matrix of size No by NA, where No represents the number of sensor objects and NA represents the number of controllable actuators in the environment. In this example, the mixing module 451 is configured to produce the actuator control signals 310 based at least in part on the AAM 440 and the environment and actuator data 004. In some examples, the mixing module 451 may be configured to produce the actuator control signals 310 based at least in part on the optional tenderer configuration dataD24100WO01452. According to some examples, the mixing module 451 may be configured to produce the actuator control signals 310 based at least in part on sensory object metadata — which may be received as part of the object-based sensory data 005, as shown in Figure 4 — such as mixing and panning laws associated with at least one sensory object.

[0094] In some examples, the mixing module 451 may be configured to produce the actuator control signals 310 based at least in part on one or more of the following:1. Thresholding elements of the AAM 440;2. Taking the maximum of a particular column of the AAM 440 — in other words, taking the sensory object that activated a particular actuator the most as the output;3. Taking any combination of the top N and performing at least one of the following:o Mixing the objects together in the actuator channel;o Pushing objects to adjacent channels.

[0095] In some embodiments, the projection module 450 may be configured to generate a sensory object image using the sensory object’s spatial coordinates — for example, x,y,z coordinates — and the sensory object’s size to produce In(x,y,z), where Inrepresents the sensory object image for the nthsensor object. Then, in some examples, the projection module 450 may be configured to compute, for every column (actuator index) of the AAM 440, the nthrow (object index), by taking the inner product of this object image and the actuator map corresponding to that actuator. The object image, actuator map and dot product may be produced and performed by the projection module 450 in any spatial domain that is convenient including, without limitation, polar, cylindrical or rectangular coordinate systems.

[0096] Some implementations may involve implementing what may be referred to herein as “repellers,” which may be used to avoid potentially undesirable sensory effects, such as lighting effects, which may be caused when sensory objects are positioned in one or more areas of a playback environment. In some such examples, repeller data may be included with the environment and actuator data 004 and the AM 423, and therefore may be part of the information that is input to the projection module 450. In some such examples, the spatial coordinate of the MS objects will be augmented when projecting them onto the AM 423 to produce the AAM 440.D24100WO01Multi-Sensory Rendering Synchronization

[0097] Object-based MS rendering involves different modalities being rendered flexibly to the endpoint / playback environment. Endpoints have differing capabilities according to various factors, including but not limited to the following:• The number of actuators,• The modalities of those actuators (e.g., light fixture vs. air flow control device vs.haptic device);• The types of those actuators (e.g., a white smart light vs. a RGB smart light, or a haptic vest vs. a haptic seat cushion) and• The location / layout of those actuators.

[0098] In order to render object-based sensory content to any endpoint, some processing of the object signals, e.g. intensities, colors, patterns etc., will generally need to be done. The processing of each modality’s signal path should not alter the relative phase of certain features within the object signals. For example, suppose that a lightning strike is presented in both the haptics and lightscape modalities. The signal processing chain for the corresponding actuator control signals should not result in a time delay of either type of sensory object signal — haptic or light — sufficient to alter the perceived synchronization of the two modalities. The level of required synchronization may depend on various factors, such as whether the experience is interactive and what other modalities are involved in the experience. Maximum time difference values may, for example, range from approximately 10ms to 100ms, depending on the particular context.HAPTICSRendering of object-based haptics content

[0099] Object-based haptics content conveys sensory aspects of the scene through an abstract sensory representation rather than a channel-based scheme only. For example, instead of defining haptics content as a single-channel time-dependent amplitude signal only, that is in turn played out of a particular haptics actuator such as a vibro-tactile motor in a vest the user wears, object-based haptics content may be defined by the sensations that it is intended to convey. More specifically, in one example, we may have a haptic object representing a collision haptic sensory effect. Associated with this object is:• The haptic object’s spatial location;D24100WO01• The spatial direction / vector of the haptic effect;• The intensity of the haptic effect;• Haptic spatial and temporal frequency data; and• A time-dependent amplitude signal.

[0100] According to some examples, a haptic object of this type may be created automatically in an interactive experience such as a video game, e.g. in a car racing game when another car hits a player’s car from behind. In this example, the MS renderer will determine how to render the spatial modality of this effect to the set of haptic actuators in the endpoint. In some examples, the Tenderer does this according to information about the following;• The type(s) of haptic devices available, e.g., haptic vest vs. haptic glove vs. haptic seat cushion vs. haptic controller;• The locale of each haptic device with respect to the user(s) (some haptic devices may not be coupled to the user(s), e.g., a floor- or seat-mounted shaker);• The type of actuation each haptic device provides, e.g. kinesthetic vs. vibro-tactile;• The on- and off-set delay of each haptic device (in other words, how fast each haptic device can turn on and off);• The dynamic response of each haptic device (how much the amplitude can vary); • The time- frequency response of each haptic device (what time-frequencies the haptic device can provide):• The spatial distribution of addressable actuators within each haptic device: for example, a haptic vest may have dozens of addressable haptics actuators distributed over the user’s torso; and• The time-response of any haptic sensors used to render closed-loop haptic effects (e.g., an active force-feedback kinesthetic haptic device.

[0101] These attributes of the haptics modality of the endpoint will inform the render how best to render a particular haptic effect. Consider the car crash effect example again. In this example, a player is wearing a haptic vest, a haptic arm band and haptic gloves. According to this example, a haptic shockwave effect is spatially located at the place where the car has collided into the player. The shockwave vector is dictated by the relative velocity of the player’ s car and the car that has hit the player. The spatial and temporal frequency spectra of the shockwave effect are authored according to the type of material the virtual cars areD24100WO01intended to be made of, amongst other virtual world properties. The Tenderer then renders this shockwave through the set of haptics devices in the endpoint, according to the shockwave vector and the physical location of the haptics devices relative to the user.

[0102] The signals sent to each specific actuator are preferably provided so that the sensory effect is congruent across all of the (potentially heterogenous) actuators available. For example, the Tenderer may not render very high frequencies to just one of the haptic actuators (e.g., the haptic arm band) due to capabilities lacking in other actuators.Otherwise, as the shockwave moves through the player’s body, because the haptic vest and haptic gloves the user is wearing do not have the capability to render such high frequencies, there would a degradation of the haptic effect perceived by the user as the wave moves through the vest, into the arm band and finally into the gloves.

[0103] Some types of abstract haptic effects include:• Shockwave effects, such as described above;• Barrier effects, such as haptic effects which are used to represent spatial limitations of a virtual world, for example in a video game. If there are kinesthetic actuators on input devices (e.g., force feedback on a steering wheel or joystick), either active or resistive, then rendering of such an effect can be done through the resistive force applied to the users input. If no such actuators are available in the endpoint then in some examples vibro-tactile feedback may be rendered that is congruent with the collision of the in-game avatar with a barrier;• Presence, for example to indicate the presence of a large object approaching the scene such as a train. This type of haptic effect may be rendered using a low timefrequency rumbling of some haptic devices’ actuators. This type of haptic effect may also be rendered through contact spatial feedback applied as pressure from aircuffs;• User interface feedback, such as clicks from a virtual button. For example, this type of haptic effect may be rendered to the closest actuator on the body of the user that performed the click, for example haptic gloves that the user is wearing.Alternatively, or additionally, this type of haptic effect may also be rendered to a shaker coupled to the chair in which the user is sitting. This type of haptic effect may, for example, be defined using time-dependent amplitude signals. However,D24100WO01such signals may be altered (modulated, frequency-shifted, etc.) in order to best suit the haptic device(s) that will be providing the haptic effect:• Movement. These haptic effects are designed so that the user perceives some form of motion. These haptic effects may be rendered by an actuator that actually moves the user, e.g. a moving platform / seat. In some examples, an actuator may provide a secondary modality (via video, for example) to enhance the motion being rendered; and• Triggered sequences. These haptic effects are characterized mainly by their timedependent amplitude signals. Such signals may be rendered to multiple actuators and may be augmented when doing so. Such augmentations may include splitting a signal in either time or frequency across multiple actuators. Some examples may involve augmenting the signal itself so that the sum of the haptic actuator outputs does not match the original signal.Spatial and Non-Spatial Effects

[0104] Spatial effects are those which are constructed in a way that convey some spatial information of the multi-sensory scene being rendered. For example, if the playback environment is a room, a shockwave moving through the room would be rendered differently to each haptic device given its location within the room, according to the position and size of one or more haptic objects being rendered at a particular time.

[0105] Non-spatial effects may, in some examples, target particular locations on the user regardless of the user’s location or orientation. One example is a haptic device that provides a swelling vibration on the users back to indicate immediate danger. Another example is a haptic device that provides a sharp vibration to indicate an injury to a particular body area.

[0106] Some effects may be non-diegetic effects. Such effects are typically associated with user interface feedback, such as a haptic sensation to indicate the user completed a level or has clicked a button on a menu item. Non-diegetic effects may be either spatial or non-spatial.Haptic Device Type

[0107] Receiving information regarding the different types of haptics devices available at the endpoint enables the Tenderer to determine what kinds of perceived effects and rendering strategics arc available to it. For example, local haptics device data indicating that the userD24100WO01is wearing both haptic gloves and a vibro-tactile vest — or at least local haptics device data indicating that that haptic gloves and a vibro-tactile vest are present in the playback environment — allows the Tenderer to render a congruent recoil effect across the two devices when a user shoots a gun in a virtual world. The actual actuator control signals sent to the haptic devices may be different than in the situation where only a single device is available. For example, if the user is only wearing a vest, the actuator control signals used to actuate the vest may differ with regard to the timing of the onset, the maximum amplitude, frequency and decay time of the actuator control signals, or combinations thereof.Location of the Devices

[0108] Knowledge of the location of the haptics devices across the endpoint enables the Tenderer to render spatial effects congruently. For example, knowledge of the location of the shaker motors in a lounge enables the Tenderer to produce actuator control signals to each of the shaker motors in the lounge in a way to convey spatial effects such as a shockwave propagating through the room. Additionally, knowledge of where wearable haptics devices, whilst implicit by their type, e.g. a glove is on the user’s hand, may also be used by the Tenderer to convey spatial effects in addition to non-spatial effects.Types of Actuation Provided by Haptic Devices

[0109] Haptic devices can provide a range of different actuations and thus perceived sensations. These are typically classed in two basic categories:1. vibro-tactile, e.g. vibrations; or2. Kinesthetic, e.g., resistive or active force feedback.

[0110] Either category of actuations may be static or dynamic, where dynamic effects are altered in real time according to some sensor input. Examples include a touch screen rendering a texture using a vibro-tactile actuator and a position sensor measuring the user’ s finger position(s).

[0111] Moreover, the physical construction of such actuators varies widely and affects many other attributes of the device. An example of this is the onset delay or time-frequency response that varies significantly across the following haptic device types:• Eccentric rotating mass:• Linear resonant actuator;• Piezoelectric actuator; andD24100WO01• Linear magnetic ram.

[0112] The Tenderer should be configured to account for the onset delay of a particular haptics device type when rendering signals to be actuated by the haptics devices in the endpoint.The On- and Off-Set Delays of the Haptic Devices

[0113] The onset delay of the haptic device refers to the delay between the time that an actuator control signal is sent to the device and the device’s physical response. The off-set delay refers to the delay between the time that an actuator control signal is sent to zero the output of the device and the time the device stops actuating.The Time-Frequency Response

[0114] The time-frequency response refers to the frequency range of the signal amplitude as a function of time that the haptic device can actuate at steady state.The Spatial-Frequency Response

[0115] The spatial-frequency response refers to the frequency range of the signal amplitude as a function of the spacing of actuators of a haptic device. Devices with closely-spaced actuators have higher spatial-frequency responses.Dynamic Range

[0116] Dynamic range refers to the differences between the minimum and maximum amplitude of the physical actuation.Characteristics of Sensors in Closed-Loop Haptics Devices

[0117] Some dynamic effects use sensors to update the actuation signal as a function of some observed state. The sampling frequencies, both temporal and spatial along with the noise characteristics will limit the capability of the control loop updating the actuator providing the dynamic effect.AIRFLOW

[0118] Another modality that some multi-sensory immersive experiences (MSIE) may use is airflow. The airflow may, for example, be rendered congruently with one or more other modalities such as audio, video, light-effects and / or haptics. Rather than only specializedD24100WO01(e.g. channel-based) setups for 4D experiences in cinemas which may include “wind effects,” some airflow effects may be provided at other endpoints that may typically include airflow, such as a car or a living room. Rather than a channel-based system, the airflow sensory effects may be represented as an airflow object that may include properties such as:• Spatial location;• Direction of the intended airflow effect;• Intensity / airflow speed; and / or• Air temperature.

[0119] Some examples of air flow objects may be used to represent the movement of a bird flying past. To render to the airflow actuators at the endpoint, the MS renderer 001 may be provided with information regarding:• The type of airflow devices e.g. fan, air conditioning, heating;• The position of each airflow device relative to the user' s location, or relative to an expected of the user;• The capabilities of the airflow device, e.g., the airflow device’s ability to control direction, airflow and temperature;• The level of control of each actuator, e.g., airflow speed, temperature range; and • The response time of each actuator, e.g., how long does it take to reach a chosen speed.Some Examples of Airflow Use in Different Endpoints

[0120] In a vehicle such as a car, the object-based metadata can be used to create experiences such as:• Mimicking “chills down your spine” during a horror movie or gaming piece of content with airflow down the chair;• Simulating the movement of a bird flying past; and / or• Create a gentle breeze in a seascape.

[0121] In the small enclosed space of a typical vehicle, temperature changes may be possible to achieve over relatively shorter periods of time — as compared to temperature changes in a larger environment, such as a living room environment. In one example, the MS renderer 001 may cause an increasing air temperature as a player enters a “lava level” or other hot area during a game. Some examples may include other elements, such as confettiD24100WO01in the air vents to celebrate an event, such as the celebration of a goal made by the user’s favorite football team.

[0122] In a living space or other room, airflow may be synchronized to the breathing rhythm of a guided meditation in one example. In another example, airflow may be synchronized to the intensity of a workout, with increased airflow or decreased temperature as intensity increases. In some examples, there may be relatively less control over spatial aspects during rendering. For example, many existing airflow actuators are optimized for heating and / or air conditioning rather than for providing spatially diverse sensory actuation.Combinations of Lights, Airflow and HapticsCar Examples

[0123] In some examples, there may be a user interface on the steering wheel or on a touchscreen near or in the dashboard. According to some examples, the following actuators may be present in the car:1. Individually addressable lights, spatially distributed around the car as follows:o on the dashboard;o under the footwells;o on the doors; ando in the center console.2. Individually controllable air conditioning / heating outlets distributed around the car as follows:o In the front dashboard;o Under the footwells;o In the center console facing the rear seats;o On the side pillars;o In the seats; ando Directed to the windscreens (for defogging).3. Individually controllable seats with vibro-tractile haptics; and4. Individually controllable floor mats with vibro-tactile haptics.

[0124] In this example, the modalities supported by these actuators include the following:• Lights across the individually addressable LEDs in the car, plus the indicator lights on the dash and steering wheel;• Air flow via the controllable air conditioning vents:D24100WO01• Haptics, including:o Steering wheel: tactile vibration feedback;o Dash touchscreen: tactile vibration feedback and texture rendering; and o Seats: tactile vibrations and movement.

[0125] In one example, a live music stream is being rendered to four users sitting in four different seat positions. In this example, the MS renderer 001 attempts to optimize the experience for multiple viewing positions. During the build-up before the artist has taken the stage and the previous acts have finished, the content contains:• Interlude music;• Low intensity lighting; and• Haptic content representing the moshing of the crowd.

[0126] In addition to the rendered audio and video stream, the light content contains ambient light objects that are moving slowly around the scene. These may be rendered using one of the ambient layer methods disclosed herein, for example such that there is no spatial priority given to any user’s perspective. In some examples, the haptic content may be spatially concentrated in the lower time-frequency spectrum and may be rendered only by the vibro-tactile motors in the floor mats.

[0127] According to this example, pyrotechnic events during the music stream correspond to multi-sensory-sensory content including:• Light objects that spatially correspond to the location of the pyrotechnics at the event; and• Haptic objects to reinforce the dynamism of the pyrotechnics via a shockwave effect.

[0128] In this example, the MS renderer 001 renders both the light objects and the haptic objects spatially. Light objects may, for example, be rendered in the car such that each person in the car perceives the light objects to come from the left if the pyrotechnics content is located at the left of the scene. In this example, only lights on the left of the car are actuated. Haptics may be rendered across both the seats and floor mats in a way that conveys directionality to each user individually.

[0129] At the end of the concert the pyrotechnics are present in the audio content and both pyrotechnics and confetti are present in the video content. In addition to rendering lightD24100WO01objects and haptic objects corresponding to the pyrotechnics as above, the effect of the confetti firing may be rendered using the airflow modality. For example, the individually controllable air flow vents of the HVAC system may be pulsed.Living Room Examples

[0130] In this implementation, in addition to an audio / visual (AV) system that includes multiple loudspeakers and a television, the following actuators and related controls are available in the living room:• A haptics vest that the user — also referred to as a player — is wearing;• Haptics shakers mounted to the seat in which the player is sitting;• A (haptics) controllable smart watch;• Smart lights spatially distributed around the room;• A wireless controller; and• An addressable air-flow bar (AFB), which includes an array of individually controllable fans directed to the user (similar to HVAC vents in the front dashboard of a car).

[0131] In this example, the user is playing a first person shooter game and the game contains a scene in which a destructive hurricane moves through the level. As it does so, ingame objects are thrown around and some hit the player. Haptics objects rendered by the MS renderer 001 cause a shockwave effect to be provided through all of the haptics devices that the user can perceive. The actuator control signals sent to each device may be optimized according to the intensity of the impact of the in-game objects, the direction(s) of the impact and the capabilities and location of each actuator (as described earlier).

[0132] At a time before the user is struck by an in-game object, the multi-sensory content contains a haptic object corresponding to a non-spatial rumble, one or more airflow objects corresponding to directional airflow; and one or more light objects corresponding to lightning. The MS renderer 001 renders the non-spatial rumble to the haptics devices. The actuator control signals sent to each haptics device may be rendered such that the ensemble of actuator control signals across the haptics array is congruent in perceived onset time, intensity and frequency. In some examples, the frequency content of the actuator control signals sent to the smart watch may be low-pass filtered, so that they are congruent with the frequency-limited capability of the vest, which is proximate to the watch. The MS renderer 001 may render the one or more airflow objects to actuator control signals for the AFB suchD24100WO01that the air flow in the room is congruent with the location and look direction of the player in the game, as well as the hurricane direction itself. Lightning may be rendered across all modalities as (1) a white flash across lights that are located in suitable locations, e.g., in or on the ceiling; and (2) an impulsive rumble in the user’s wearable haptics and seat shaker.

[0133] When the user is struck by an in-game object, a directional shockwave may be rendered to the haptics devices. In some examples, a corresponding airflow impulse may be rendered. According to some examples, a damage take effect, indicating the amount of damage caused to the player by being struck by the in-game object, may be rendered by the lights.

[0134] In some such examples, signals may be rendered spatially to the haptics devices such that a perceived shockwave moves across the player’s body and the room. The MS renderer 001 may provide such effects according to actuator location information indicating the haptics devices locations relative to one another. The MS renderer 001 may provide the shockwave vector and position according to the actuator location information in addition to actuator capability information. According to some examples, a non-directional air flow impulse may be rendered, e.g., all the air vents of the AFB may be turned up briefly to reinforce the haptic modality. In some examples, at the same time, a red vignette may be rendered to the light strip surrounding the TV, indicating to the player that the player took damage in the game.

[0135] Figure 5 shows example elements of another system for the creation and playback of MS experiences. As with other figures provided herein, the types and numbers of elements shown in Figure 5 are merely provided by way of example. Other implementations may include more, fewer and / or different types and numbers of elements. According to some examples, system 500 may be, or may include, one or more devices configured for performing at least some of the methods disclosed herein. In some examples, system 500 may include one or more instances of the control system 110 of Figure 1 A that are configured for performing at least some of the methods disclosed herein.

[0136] According to this example, the system shown in Figure 5 is an instance of the system shown in Figure 3. In this example, the system shown in Figure 5 is a “lightscape” embodiment in which video, audio and light effects are combined to create the MS experience.D24100WO01

[0137] In this example, system 500 includes a lightscape creation tool 100, which is an instance of the content creation tool 000 that is described with reference to Figure 3. The lightscape creation tool 100 is configured for designing and outputting object-based light data 505’, either separately or in conjunction with corresponding audio data 111’ and / or video data 112’, depending on the particular implementation. The object-based light data 505’ may include time stamp information, as well as information indicating light object properties, etc. In some instances, the time stamp information may be used to synchronize effects relating to the object-based light data 505’ with the audio data 111’ and / or the video data 112’, which also may include time stamp information.

[0138] In this example, the object-based light data 505’ includes light objects and corresponding light metadata. For example, the object-based light data may include light object position metadata, light object color metadata, light object size metadata, light object intensity metadata, light object shape metadata, light object diffusion metadata, light object gradient metadata, light object priority metadata, light object layer metadata, or combinations thereof. Although the content creation tool 100 is shown providing a stream of object-based light data 505’ to the experience player 102 in this example, in alternative examples the content creation tool 100 may produce object-based light data 505’ that is stored for subsequent use. Examples of graphical user interfaces for a light-object-based content creation tool are described below.

[0139] In this example, system 500 includes an experience player 102 that is configured to receive object-based light data 505’, audio data 111’ and video data 112’, and to provide object -based light data 505 to the lightscape Tenderer 501, to provide audio data 111 to the audio Tenderer 106 and to provide video data 112 to the video Tenderer 107. As noted elsewhere herein, the object-based light data 505, the audio data 111 and the video data 112 may include time stamp information that may be used to synchronize MS effects with audio and / or video effects. According to some examples, the experience player 102 may be a media player, a game engine or personal computer or mobile device, or a component integrated in an television, DVD player, sound bar, set top box, or a service provider media device such as a Chromecast, Apple TV device, or Amazon Fire TV. In some examples, the experience player 002 may be configured to receive encoded object-based light data 505’ along with encoded audio data 111' and / or encoded video data 112’, e.g., as part of the same bitstream with the encoded audio data 111’ and / or the encoded video data 112’. According to some examples, the experience player 102 may be configured to extract the object-basedD24100WO01light data 505 from the content bitstream and to provide decoded object-based light data 505 to the lightscape Tenderer 501, to provide decoded audio data 111 to the audio Tenderer 106 and to provide decoded video data 112 to the video Tenderer 107. In some examples, the experience player 002 may be configured to allow control of configurable parameters in the lightscape Tenderer 501, such as immersion intensity. Some examples are described below.

[0140] As noted above, according to some implementations the system 500 may include one or more instances of the control system 110 of Figure 1 A configured for performing at least some of the methods disclosed herein. In some such examples, one instance of the control system 110 may implement the lightscape creation tool 100 and another instance of the control system 110 may implement the experience player 002. In some examples, one instance of the control system 110 may implement the audio Tenderer 006, the video Tenderer 007, the lightscape Tenderer 501, or combinations thereof. According to some examples, an instance of the control system 110 that is configured to implement the experience player 002 may also be configured to implement the audio Tenderer 006, the video Tenderer 007, the lightscape Tenderer 501, or combinations thereof.

[0141] In some examples, room descriptors of the environment and light fixture data 104 may describe the size and orientation of the playback environment itself, to establish a relative or absolute coordinate system to which all objects are positioned. Room descriptor information may indicate or describe the physical dimensions of the playback environment, for example in physical units of distance such as meters. In some such examples, sensory object locations, sensory object sizes, and sensory object orientations may be described in units that are relative to the room size, for example in a range from -1 to 1. Room descriptors may also describe a preferred viewing position. For example, in a living room a display screen may be regarded as the front, in some instances the front and center, and the floor and ceiling may be regarded as the vertical bounds. In some such examples, the room descriptors also may also indicate bounds correspond! ng with the left, right, front, and rear, walls relative to the front position. According to some examples, at least some room descriptor information may be provided as a matrix. In some such examples the matrix may be a 3x3 matrix, with one row or column corresponding to one dimension of a three-dimensional space.

[0142] According to this example, system 500 includes a lightscape Tenderer 501 that is configured to render object-based light data 505 to light fixture control signals 515, based atD24100WO01least in part on environment and actuator data 104. In this example, the lightscape Tenderer 501 is configured to output the light fixture control signals 515 to light controllers 103, which are configured to control the light fixtures 108. The light fixtures 108 may include individual controllable light sources, groups of controllable light sources (such as controllable light strips), or combinations thereof. In some examples, the lightscape Tenderer 501 may be configured to manage various types of light object metadata layers, examples of which are provided herein. According to some examples, the lightscape Tenderer 501 may be configured to render actuator signals for light fixtures based, at least in part, on the perspective of a viewer. If the viewer is in a living room, that includes a television (TV) screen, the lightscape tenderer 501 may, in some examples, be configured to render the actuator signals relative to the TV screen. However, in virtual reality (VR) use cases, the lightscape tenderer 501 may be configured to render the actuator signals relative to the position and orientation of the user’s head. In some examples, the lightscape tenderer 501 may receive input from the playback environment — such as light sensor data corresponding to ambient light, camera data corresponding to a person’s location or orientation, etc. — to augment the render.

[0143] In some examples, the lightscape tenderer 501 is configured to receive object-based light data 505 that includes light objects and object-based lighting metadata indicating an intended lighting environment, as well as environment and light fixture data 104 corresponding to light fixtures 108 and other features of a local playback environment, which may include, but are not limited to, reflective surfaces, windows, non-controllable light sources, light-occluding features, etc. In this example, the local playback environment includes one or more loudspeakers 109 and one or more display devices 510.

[0144] According to some examples, the lightscape tenderer 501 is configured to calculate how to excite various controllable light fixtures 108 based at least in part on the object-based light data 505 and the environment and light fixture data 104. The environment and light fixture data 104 may, for example, indicate the geometric locations of the light fixtures 108 in the environment, light fixture type information, etc. In some examples, the lightscape tenderer 501 may be configured to determine which light fixtures will be actuated based, at least in part, on the position metadata and size metadata associated with each light object, e.g., by determining which light fixtures are within a volume of a playback environment corresponding to the light object's position and size at a particular time indicated by light object time stamp information. In this example, the lightscape tenderer 501 is configured toD24100WO01send light fixture control signals 515 to the light controller 103 based on the environment and light fixture data 104 and the object-based light data 505. The light fixture control signals 515 may be sent via one or more of various transmission mechanisms, application program interfaces (APIs) and protocols. The protocols may, for example, include Hue API, LIFX API, DMX, Wi-Fi, Zigbee, Matter, Thread, Bluetooth Mesh, or other protocols.

[0145] In some examples, the lightscape Tenderer 501 may be configured to determine a drive level for each of the one or more controllable light sources that approximates a lighting environment intended by the author(s) of the object-based light data 505. According to some examples, the lightscape Tenderer 501 may be configured to output the drive level to at least one of the controllable light sources.

[0146] According to some examples, the lightscape Tenderer 501 may be configured to collapse one or more parts of the lighting fixture map according to the content metadata, user input (choosing a mode), limitations and / or configuration of the light fixtures, other factors, or combinations thereof. For example, the lightscape Tenderer 501 may be configured to render the same control signals to two or more different lights of a playback environment. In some such examples, two or more lights may be located close to one another. For example, two or more lights may be different lights of the same actuator, e.g., may be different bulbs within the same lamp. Rather than compute a very slightly different control signal for each light bulb, the lightscape Tenderer 501 may be configured to reduce the computational overhead, increase rendering speed, etc., by render the same control signals to two or more different, but closely-spaced, lights.

[0147] In some examples, the lightscape Tenderer 501 may be configured to spatially upmix the object-based light data 505. For example, if the object-based light data 505 was produced for a single plane, such as a horizontal plane, in some instances the lightscape Tenderer 501 may be configured to project light objects of the object-based light data 505 onto an upper hemispherical surface (e.g., above an actual or expected position of the user’s head) in order to enhance the experience.

[0148] According to some examples, the lightscape Tenderer 501 may be configured to apply one or more thresholds, such as one or more spatial thresholds, one or more luminosity thresholds, etc., when rendering actuator control signals to light actuators of a playback environment. Such thresholds may, in some instances, prevent some light objects from causing the activation of some light fixtures.D24100WO01

[0149] In some implementations, the lightscape Tenderer 501 may be configured to adapt to changing conditions. Some examples of lightscape Tenderer 501 implementations are described in more detail below.

[0150] Light objects may be used for various purposes, such as to set the ambience of the room, to give spatial information about characters or objects, to enhance special effects, to create a greater sense of interaction and immersion, to shift viewer attention, to punctuate the content, etc. Some such purposes may be expressed, at least in part, by a content creator according to sensory object metadata types and / or properties that are generally applicable to various types of sensory objects — such as object metadata indicating a sensory object’s location and size.

[0151] For example, the priority of sensory objects, including but not limited to light objects, may be indicated by sensory object priority metadata. In some such examples, sensory object priority metadata is taken into account when multiple sensor objects map to the same fixture(s) in a playback environment at the same time. Such priority may be indicated by light priority metadata. In some examples, priority may not need to be indicated via metadata. For example, the MS renderer 001 may give priority to sensory objects — including but not limited to light objects — that are moving over sensory objects that are stationary.

[0152] A light object may, depending on its location and size and the locations of light fixtures within a playback environment — potentially cause the excitation of multiple lights. In some examples, when the size of a light object encompasses multiple lights, the Tenderer may apply one or more thresholds — such as one or more spatial thresholds or one or more luminosity thresholds — to gate objects from activating some encompassed lights.Examples of Using a Lighting Map

[0153] In some implementations a lighting map, which is an instance of the of the actuator map (AM) that includes a description of lighting in a playback environment, may be provided to the lightscape Tenderer 501. In some such examples, the environment and light fixture data shown in Figure 5 may include the lighting map. According to some examples, the lighting map may be allocentric, e.g., indicating absolute spatial coordinate-based light fall-off, whereas in other examples the lighting map may be egocentric, e.g., a light projection mapped onto a sphere at an intended viewing position and orientation. In the caseD24100WO01of a sphere, the lighting map may, in some examples, be projected onto a two-dimensional (2D) surface, e.g., in order to utilize 2D image textures in processing. In any case, the lighting map should indicate the capabilities and the lighting setup of the playback environment, such as a room. In some embodiments the lighting map may not directly relate to physical room characteristics, for example if certain user preference-based adjustments have been made.

[0154] In some examples, there may be one lighting map per light fixture, or per light, in a playback environment. According to some examples, the intensity of light indicated by the light map may be inversely correlated to the distance to the center of the light, or may be approximately (e.g., within plus or minus 5%, within plus or minus 10%, within plus or minus 15%, within plus or minus 20%, etc.) inversely correlated to the distance to the center of the light. The intensity values of the light map may indicate the strength or impact of the light object onto the light fixture. For example, as a light object approaches a lightbulb, the lightscape Tenderer 501 may be configured to determine that the lightbulb intensity will increase as the distance between the light object and the lightbulb decreases. The lightscape Tenderer 501 may be configured to determine the rate of this transition based, at least in part, on the intensity of light indicated by the light map.

[0155] Figure 6A shows an example light map for a table lamp. The area 602 indicates the location of the lamp, mapped onto polar coordinates from the main viewer position. The lightscape Tenderer 501 is, in some examples, configured to map lightscape objects into a common rendering space, which may be allocentric or egocentric. Figure 6B shows an example of an egocentric light map. In this example, Figure 6B shows mapping for a spot light object onto a sphere at an intended viewing position and orientation. In Figures 6A and 6B, darker areas are indicated by dots that are relatively closer together, whereas brighter areas are indicated by dots that are relatively farther apart.

[0156] Inside this common rendering space, in some examples the lightscape Tenderer 501 may be configured to use a dot product multiplication between a light object and the light map for each light to compute a light activation metric, e.g., as follows:LM ■ Objmin(£ LM, £ Obj)

[0157] In the foregoing equation, Y represents the light activation metric, LM represents the lighting map and Obj represents the map of a light object. The light activation metricD24100WO01indicates the relative light intensity for the actuator control signal output by the lightscape Tenderer 501 based on the overlap between the light object and the spread of light from the light fixture. In some examples, the lightscape Tenderer 501 may use the maximum or closest distance, or other geometric metrics, from the light object to the light fixture as part of the determination of light intensity. In some implementations, instead of computing the light activation metric, the lightscape Tenderer 501 may refer to a look-up-table to determine the light activation metric.

[0158] The lightscape Tenderer 501 may repeat one of the foregoing procedures for determining the light activation metric for all light objects and all controllable lights of the playback environment. Thresholding for light objects that produce a very low impact on light fixtures may be helpful to reduce complexity. For example, if the effect of a light object would cause an activation of less than a threshold percent of light fixture activation — such as less than 10%, less than 5%, etc. — the lightscape Tenderer 501 may disregard the effect of that light object.

[0159] The lightscape Tenderer 501 may then use the resultant light activation matrix Y, along with various other properties such as the chosen panning law (either indicated by light object metadata or Tenderer configuration) or the priority of the light object, to determine which objects get rendered by which lights and how. Rendering lights-objects into light fixture control signals may involve:• Altering the luminance of a light-object as a function of the distance it is from the light fixture;• Mixing the colors of multiple light-objects that are simultaneously (multiplexed) rendered by a single light fixture: or• Altering either of the above based on the light object priority.Some detailed examples are disclosed herein.RENDERING PARAMETERS

[0160] In addition to the information carried by the light object metadata, the rendering of light-objects can be a function of the settings or parameters of the lightscape Tenderer 501 itself. These may include:D24100WO01• Velocity priority - when this parameter is set, light objects that are moving are given a higher priority than those which are not. Having the velocity priority parameter set enhances the dynamism of the rendered scene;• Color priority - light-objects with higher saturation values will take priority;• Activation threshold - the minimum light activation, Y, that must be achieved in order to activate a light-fixture;• Accessibility - certain colors may be chosen over others to best represent the experience for colorblind users. Certain flash rates may be avoided for those with photo-sensitivities.RENDERING CONFIGURATION (MODES)

[0161] In addition to the information carried by the light object metadata, the lightscape Tenderer 501 may, in some implementations, be configured according to different modes. As used herein, the term “mode” is different from “parameter” in the sense that modes may, for example, involve completely different signal paths, whereas parameters may simply parameterize these signal paths. For example, one mode may involve the projection of all light objects onto a lighting map before determining how / what to render to the light-fixtures, while another mode may only snap the highest-priority lights to the nearest light fixtures. Modes may include:• Modes to support low light-fixture count. In these modes, the rendering parameters and the light object metadata are utilized in order to determine which subset of lightobjects are to be rendered and in what manner. Here, the “manner” refers to the tradeoff between the spatial, color, temporal fidelity of the most prominent light-objects in the scene;• Modes to support different content types, such as music vs. gaming;• Modes in which multiple light objects may be rendered by a single light fixture (or a single light) with color mixing;• Modes in which only a single light object can be rendered by a single light fixture (or a single light);• Modes in which the luminance of the light object is altered as a function of the geometric - or otherwise - distance between the light object and light fixture.D24100WO01COLOR MIXING AND PRIORITIZATION

[0162] Some implementations of the lightscape Tenderer 501 may implement one or more color mixing methods, prioritization methods, or combinations thereof. For example, when there are multiple light objects that simultaneously influence the same light fixture(s), the lightscape Tenderer 501 may implement a color prioritization algorithm. In the simplest embodiment, only one light object influences the light fixture. The particular light object that will affect the light fixture may, in some examples, be determined by one of the following criteria: (1) the light object that is closest to the light fixture, which may be referred to as “snap-to-color”; (2) the light object that has the highest percentage of its impact on that light fixture, (3) the light object with the highest priority pre-defined, (4) the light object that is the brightest. In each of these conditions, only the reference color (potentially at a diminished brightness) may be shown in some instances.

[0163] In some instances it may be desirable to mix lighting. For example, when mimicking the physical characteristics of having multiple colored light fixtures with a single light fixture, modeling the physics of light mixing can make rendering more realistic. This is best done in a physics based, perceptually uniform, color space such as XYZ. In some examples, light mixing may be a linear process within a color space. According to some examples, when there are multiple light objects that simultaneously influence the same light fixture(s), light mixing may involve mixing color and adding intensity.

[0164] According to some examples, the lightscape Tenderer 501 may be configured to calculate the light mixing of two light objects as follows:XYZnew= a * XYZ1+ β * XYZ2

[0165] In this example, the mixing occurs in the XYZ color space. In the foregoing expression, XYZnewrepresents the result of light mixing, XYZ\ represents the color of a first light object, a represents a constant that indicates the weighting of first light object’s color, XYZ2represents the color of a second light object and represents a constant that indicates the weighting of second light object’s color. The alpha and beta values may, for example, correspond to the amount of light intensity falloff due to the distance from each light object to the light fixture. The alpha and beta values may, for example, be extracted from the lighting map or from some other, potentially geometric, model.D24100WO01

[0166] In cases where physical modeling may be too computationally expensive, faster methods will allow for approximations. In one embodiment, the lightscape Tenderer 501 may be configured to calculate the light mixing of two light objects in the HSV color space, as follows:HSVnew= [a * HS1+ β * HS2, V1+ V2]In the foregoing expression, H represents hue and S represents saturation, HS1represents the color of a first light object, a represents a constant that indicates the weighting of first light object’s color, Vi represents the intensity of the first light object, HS2represents the color of a second light object, P represents a constant that indicates the weighting of second light object’s color and V2 represents the intensity of the second light object. In this example, the hue (H) and the saturation (S) are scaled by alpha and beta based on the amount they contribute at the rendered light fixture location. Then the intensities (V) are added to model the addition of light sources.

[0167] A typical lightscape scene may have anywhere from a few light objects up to a few dozen light objects that are active at any given time. A content creator will generally want to have control over the way that these light objects interact with one another within the lightscape Tenderer 501 as they are rendered onto the light fixtures in the endpoint. This may be summarized as the creator wanting to control (1) the relative priority of light objects and (2) the way in which properties of light objects can and cannot be mixed together. The latter is the larger departure from object-based audio rendering procedures, because the lightscape Tenderer 501 normally cannot simply mix the effects of multiple light objects to compute actuator control signals for one actuator. When there are a large number of light objects (even just 3), in combination with the projection of color from the object spatial domain onto the light-fixtures which introduces some warping, the color resulting from simply mixing the effects of multiple light objects would generally not represent the creative intent or preserve the fidelity of the scene. Furthermore, we are restricted to the limited output capacity of the light-fixtures in the endpoint.

[0168] The foregoing issues highlight the importance of providing the content creator with some control over the actions of the lightscape Tenderer 501. Various disclosed examples provide the content creator with the ability to define the light objects layer, the priority on that layer, to control how light objects are mixed on a layer, how layers are mixed, or combinations thereof.D24100WO01

[0169] Figure 7 shows elements of a lightscape renderer according to some examples. As with other figures provided herein, the types and numbers of elements shown in Figure 7 are merely provided by way of example. Other implementations may include more, fewer and / or different types and numbers of elements. According to this example, the lightscape tenderer 501 is an instance of the lightscapc tenderer 501 that is described with reference to Figure 5. In some examples, the lightscape tenderer 501 may be implemented by one or more instances of the control system 110 of Figure 1 A.

[0170] According to this example, the lightscape tenderer 501 includes the following elements:• 705: Light objects, which are instances of the object-based sensory data 005 disclosed herein;• 004: Environment and actuator data;• 723: A lighting map (LM), which is an instance of the of the actuator map (AM) that includes a description of lighting in a playback environment;• 750: A projection module, which is an instance of the projection module 450 of Figure 4 and is configured to projects the light objects using the LM 723;• 740: A light activation matrix (LAM), which is an instance of the actuator activation matrix (AAM) 440 and is the output of the projection module 750;• 751: A mixing module 751, which is an instance of the mixing module 451 of Figure 4 and is configured to convert the LAM 740 into actuator commands 741;• 741: The actuator commands 741, which in this example are sent directly to the light fixtures — which are instances of the actuators 008 — to control them, but which may in other examples be sent to light controller APIs 103, which will send corresponding control signals to the light fixtures 008;• 752: Renderer configuration data, which may include settings such as the desired dynamism and mode. In some disclosed context-aware examples, the renderer configuration data 752 may be changed automatically;• 702: An intra-layer mixing module, which is configured to mix light objects of the same layer according to a mixing law;• 710: Light activation vectors, in this example a light activation vector 710 for every layer, containing the activation values and mixed colors of every light fixture; and • 703: An inter-layer blending module, which is configured to blend the light activation vectors 710 together to obtain the rendered actuator commands 741.D24100WO01

[0171] According to some examples, the light (actuator) activation matrix (LAM) 740 is a real matrix of size Noby NA, which is denoted as A in the following equation:ao,oa0, NA-l.alVo-l,0 ■"aNo-l, NA-l.In the foregoing equation, ai,j represents the activation value of the ithlight object on the jthactuator. The actuators are lights in this example. The ithrow of the A matrix contains all of the lights activated by the ithlight object. The jthcolumn of the A matrix represents all of the light objects activating the jthlight fixture. This is a useful intermediate data product, as the lightscape Tenderer 501 has not yet performed any mixing of light objects, which results in information loss. This has various potential benefits, including the possibility of optimizing the scene by analyzing A and then warping the scene (see examples below).

[0172] In some examples, there may be an A matrix for every layer being processed by the lightscape Tenderer 501. Thus, the lightscape Tenderer 501 can define a tensor of size Noby by Nt(dimension ordering is arbitrary), where Ntrepresents the number of layers in the lightscape Tenderer 501. In this document A may refer to either the tensor or the matrix form. The context will inform the reader which it is.

[0173] According to some examples, the intra-layer mixing module 702 is configured to mix all of the light objects on a given layer. This process collapses the A matrix (for that layer) into a light activation vector 710 v, of size 1 by NA. The A matrix contains the activation values, not the color values. We can express the intra-layer mixing module 702 in its general form as a function which produces the light activation vector v (710), as follows:v - A (4)

[0174] However, in this example the intra-layer mixing process also produces a vector c for every layer in the matrix containing the color mixing result for that layer, so we may alter the equation above as follows:v, c = (A c0)

[0175] In the above equation, AO now outputs both the activation vector, v, and the intralayer mixed colors c as a function of A and c0, which is a vector of length Nocontaining all of the objects’ colors.D24100WO01

[0176] The interlayer blending process takes all Ntv and c vectors packed into v’ and c’ matrices of size NAby Ntand outputs a single vector o of length NAcontaining the colors for each actuator. This can be written generally as:o=

[0177] Examples of () are given in the Example Object Mixing Laws section of this disclosure. Examples of g () are given in the Example Layer Blending Laws section of this disclosure.Example Light Activation Laws

[0178] Light activation laws can range from simple geometric projections and distances to complex responses that involve precomputing and storing in a look up table (LUT). An example of the latter is the lighting map. In some examples, the LUT provides an activation value that may be based on the objects size, position, layer, velocity and potentially other parameters. This activation value may not represent how much the actual light or light fixture projects light onto the playback environment. In some examples, the activation value may be optimized to provide a sparse A matrix (which simplifies the mixing and potential object prioritization problems) and is based on such a (measured or simulated) projection.

[0179] Simple geometric-based activation laws may be applied in different reference frames, e.g., an allocentric or an egocentric reference frame. Such activation laws may be applied using various coordinate systems, for example rectangular, spherical or cylindrical coordinate systems.

[0180] Following is an example of an allocentric rectangular activation law:ALGORITHM 1For i=0; i< No; i ++ do; / / loop over all objectsFor j=0; j< NA; j ++ do / / loop over all light-fixturesA[i,j] = 0For i=0; i< No; i ++ do; / / loop over all objectsFor j=0; j< NA; j ++ do / / loop over all light-fixturesd= ii rs-sii2D24100WO01if d <orzeA[i,j] = 1Else if d < (Opze+ o[eatfier')A[i,j] = ( 0?ize+ o[eather— d) / (o[eattier+£)Algorithm 1 may, for example, be implemented by a control system that is configured to provide an instance of the projection module 750 of Figure 7. In Algorithm 1,O[represents the ithobject;L represents the jthlight-fixture;d represents the Euclidean distance between the light and object;O?lzerepresents the size of the object in the lightscapc metadata;gfeatheirepresenis thefeather radius of the object in the lightscape metadata; and s represents a small number, e.g. 10A-10 for regularization.

[0181] An implementation of an egocentric spherical activation law may be substantially like the above Euclidean implementation, but one in which the coordinates are first transformed so that the user position and orientation define the origin (the light and object positions are references from this position and orientation) and the position and size are now angular. Distances and activation functions that can be used in place of d include, but are not limited to, the Euclidean distance, p-norm distance, cosine distance, logistic function, gaussian activation function, and rectified linear activation.

[0182] Additional variations to activations laws may include:• warping the coordinates of the objects to account for multiple user perspectives (widening the sweet spot); and / or• using arbitrary reference frames, e.g., rotated frames to account for a TV / screen that is not placed orthogonally within the playback environment.Example Object Mixing Laws

[0183] A simple implementation of f Q would be to simply sum over all Norows of A to produce v and using these to weight coto accumulate c. More specifically, for the jth light, v and c may be calculated as follows:i=No~1v[j] = A[i,j]i=0D24100WO01i = N0-lc[ / ] = 4[iJ] c0[i]i=0

[0184] Some implementations may involve performing a normalization across the columns of A before performing the summation. This normalization may be linear, e.g.,:...Z‘ZO°_1MJ]Or nonlinear, for example a SoftMax function:

[0185] However, these normalizations result in the sum of the columns becoming unity, which may not be desirable when performing inter-layer mixing. Some examples involve augmenting the SoftMax function to place it back onto the same range, so that the sum of the columns are equal after the non-linear normalization (which is now a scaling, not a norm), e.g., as follows:

[0186] The above-described scaling and / or normalization may be performed before, in some instances, only the top N object elements are contributing to the jthlight. Normalization and scaling are motivated by the fact that it may not be desirable to saturate the color c[j ]. If we consider an example involving the RGB color model on a range of [0, 1], then the mixing laws above can produce results that exceed 1. In such cases, clipping may be performed in order to send valid RGB codewords to the light fixtures. However, clipping introduces chromaticity errors and, in the worst case when all 3 RGB channels are saturated, the output is white. This is not desirable, which is a further motivator for only taking the top N (for example, N=2) when performing the summations above.

[0187] In some examples, the lightscape Tenderer 501 may be configured for screen mixing. According to some examples, screen mixing may be implemented as follows:ALGORITHM 2For j = 0; j < NA; j + +; do / / loop over all light fixtures:D24100WO01c[y] = 0v[j] = 0For i = 0; i < No; i + +; do; 11 loop over all objectsc[j]+= 1 - (1 - A[i,j]c0[i]) (1 - c[j])v[ / ]+=

[0188] According to some examples, the lightscape Tenderer 501 may be configured for screen mixing with or without activation weighting as exemplified with the screen mixing above.Example Layer Blending Laws

[0189] In some implementations, there is no fundamental difference between the mixing that happens in the intra-layer process and the blending that happens in the inter-layer process. Blending is typically used to refer to the process of combining multiple layers in image processing and computer graphics. Thus, to avoid confusion and to help delineate the intra- and inter-layer processes, the terms “mixing” and “blending” are used herein.

[0190] For blending layers, in one example we may use alpha compositing, specifically A over B alpha compositing. Recall that blending layers is the process of producing the output vector o from the matrices v’ and c’:o = g (v',c')In the foregoing equation, v’ and c’ are of size NAby Nt, for example as follows:■V0,0 ”■V0, N;-l 'v' =TN^-I. O ■”VNA-I, NI~I.■c0,0 ”■c0, W(-l ■c' =.CJV^-l,0 ”■CNA-1, NI~1.In the foregoing v’ equation, vtj represents the net activation value of the light objects mixed into the ithlight-fixture on the jthlayer. These light objects are mixed to produce the Cij color.D24100WO01

[0191] In some examples in which the lightscape Tenderer 501 uses multiple layers, the order of the layers may imply some semantics or priority. For example, in some instances there may be an ambient layer, a spatial layer and an overlay layer. In some such examples, the lightscape Tenderer 501 may render these layers in order, in other words the ambient layer may be rendered first, then the spatial layer, then the overlay layer. According to some such examples, the lightscape Tenderer 501 may implement methods such as alpha compositing to render these layers and may use the net activation values, v, as a proxy for the alpha values to blend these layers.

[0192] Following is an example of using A over B alpha compositing to implement the function g () when using an RGB color model.ALGORITHM 3=[0,0,0] / / RGB value for blackFor i = 0; i < NA; i + +; do / / loop over all light-fixtureso[i] = c[i, 0]aacc= v[i, 0]For I = 1; I < Nt; I + +; do / / loop over layersafJ= v[i, I — 1]ab= v[i, I]Ifaacc > 1-0break; / / move on to next light fixturea0= aa+ ab(l — aa)If a0<— 0o[j] (-blackelseo[i] = (aao[i] + ab(1 — aa)c[i, l])D24100WO01&acc +

[0193] In Algorithm 3, the layers are ordered so that I = 0 has the highest priority and I = Nt— 1 has the lowest.Feather Distance

[0194] A feather distance is a distance applied to the sensory objects such that spatial smoothing occurs. It is used to create smooth transitions as objects move around and activate I deactivate actuators. For example, if an object has size = 0.1 and a feather distance (sometimes also called feather size in our docs) of 0.2, then an example activation law would produce an activation value A of:( 1, d < Ofzej d ~ Ofze| Qfeather ' Ofize< d < Ofize+ ofeatherI ‘ 0, otherwised = || O?os

[0195] In the foregoing discussion, sensory objects were generally assumed to have a circular shape or a spherical shape. Some aspects of this disclosure describe sensory object areas or volumes that may have a wide variety of shapes. In some examples, the sensory object areas or volumes may be defined according to various types of parameters. The corresponding sensory objects may be referred to herein as parameterizable sensory objects. Some disclosed examples involve extended light fixtures — which are also referred to herein as extended lights — and methods to compute a light activation matrix (LAM) for parameterizable sensory objects that interact with extended lights.

[0196] Figures 8A-9C illustrate two-dimensional (2D) examples of sensory objects for the sake of simplicity. Similarly, Figures 11 A-l 1C illustrate 2D examples of sensory objects and extended lights for the sake of simplicity. However, the reader should assume that these 2D illustrations represent, or could represent, three-dimensional (3D) shapes unless the corresponding discussion explicitly states otherwise.

[0197] Parameterizable Primitives

[0198] Figure 8A shows an example of a playback environment within which are examples of a parameterizable sensory object and a light fixture. As with other figures providedD24100WO01herein, the types, number and arrangement of elements shown in Figure 8A are merely provided by way of example. Other implementations may include more, fewer and / or different types, numbers and / or arrangements of elements.

[0199] In Figure 8A, a sensory object 802 is represented in a playback environment 800 where a sensory experience is being rendered. A playback environment may also be referred to herein as an endpoint. According to this example, Figure 8A includes the following elements:800: a playback environment, which in this example is a room in which a lightscape experience is being rendered;801: a light fixture in the room 800:802: a sensory object, which is a light object having the shape of a rectangular prism in this example;803 & 804: vectors representing the length and width dimensions of the light object 802;807: the centroid of the light object 802; and810: a vector between the centroid 807 of the light object 802 and the light fixture 801.

[0200] Discrete and parameterizable spatial objects, such as the light object 802, provide simple mechanisms for the creation and rendering of sensory content. In some examples, the light object 802 has at least the following properties:Position;Shape, e.g., polyhedron, ellipsoid, bicone or cylinder;Size: andAngular orientation (attitude).

[0201] The vector 810 of Figure 8 A is a specific example of a vector d ’ between an i,hsensory object (such as the light object 802 of Figure 8A) and a fhactuator of a playback environment (such as the light fixture 801 in the room 800). The vector d’ may be computed by taking the difference in the positions of the ithsensory object and the jthactuator, e.g., as follows:-Ofos

[0202] In the foregoing expression, O?osrepresents the position of the Ithsensory object and L^osrepresents the position of the jthactuator. To determine whether a sensory object (inD24100WO01this example, a light object) activates an actuator (in this example, a light), we can rotate d'a into the reference frame of the sensory object, e.g., as follows:d'^j = L'-os- 0os

[0203] With reference to the examples shown in Figure 8A, the elements of the foregoing expressions are as follows:G [R!vrepresents the position of the light fixture 801 in the room 800 reference frame;N represents the dimensionality, such as 2D or 3D;OosG Rwrepresents the position of the light object 802 in the room 800 reference frame;d'i ■ G Rwrepresents the offset vector 810 in the room 800 reference frame; d[j G IR<Wrepresents the offset vector 810 in the light object 802 reference frame, with the origin of the offset vector 810 at the centroid 807 of the light object 802; and R Gx Nrepresents a rotation matrix that indicates the orientation of the light object 802 with respect to the room 800 reference frame.

[0204] The foregoing expressions may be with reference to 2D or 3D depending on the particular implementation, e.g., depending on the endpoint configuration and the Tenderer configuration.

[0205] An activation function act() may be used to compute A[i,j], which represents the activation function of the jth actuator by the ith sensory object. In some examples, the activation function act() may simply be a function that determines whether dtlies within a region determined by the sensory object’s size OlzeG Rw.Parameterizable Primitives with Feathering

[0206] Feathering is a parameter that defines a region, which is normally a region surrounding a sensory object, in which the activation value corresponding to the sensory object may vary from 0 (no activation) to 1 (full or 100% activation). From one edge of a feathering region to the other, the activation value may “taper off’ according to a linear or a non-linear decay function. Feathering can provide a spatial smoothing effect as sensoryD24100WO01objects move throughout a playback environment, as sensory objects actuate multiple actuators at the same time, etc.

[0207] Figure 8B shows an example of the playback environment of Figure 8 A in which the parameterizable sensory object includes feathering. As with other figures provided herein, the types, number and arrangement of elements shown in Figure 8B arc merely provided by way of example. Other implementations may include more, fewer and / or different types, numbers and / or arrangements of elements.

[0208] According to this example, Figure 8B includes the following elements:800: a playback environment, which in this example is a room in which a lightscape experience is being rendered;801: a light fixture in the room 800:802: a sensory object, which is a light object having the shape of a rectangular prism in this example;803 and 804: vectors representing the length and width dimensions of the light object 802;807: the centroid of the light object 802;810: a vector between the centroid 807 of the light object 802 and the light fixture 801;814 and 815: vectors representing a zone in which a feathering function will be applied; and817: the feathering region.

[0209] An example activation function for a N-dimensional cuboid with feathering (jfeat,ier(which may be set to zero) can be written as follows:

[0210] ALGORITHM 1Input di, Oize, oeather, NOutput activationFunction CuboidActivation, Oflze, QfeatherN)inside_count «- 0outside_count «- 0activation <- inffor n=0; n < N; n ++;D24100WO01if ( [n] < 0?ize[n]inside_count++if d^n] > (0-ize[n] + Qfeather[n])outside_count ++|diJ[n]-O^[n]|0{eatller[n]if a > 1.0 or a < 0.0continueactivation = min(a, activation)if inside_count== Nreturn 1.0if outside_count > 0return 0.0return activation

[0211] An example activation function for a N-dimensional ellipsoid can be written as follows:

[0212] ALGORITHM 2Input dtJ, 0lize,Ofeather, NOutput activationFunction EllipsoidActivation (dt, 0flze, Qfeather, N)QdiJ[n] 2O?izeini_ dij [n] _OfizeW+ + 0{eather[n])if a< 1.0return 1.0if b < 1.0D24100WO01return 1.0 - breturn 0.0

[0213] An example activation function for a N-dimcnsional cylinder (which reduces to a cuboid or a circle in 2D) can be written as Algorithm 3, below. In Algorithm 3, the main axis of the cylinder is its first, that is Olze[0] is the height of the cylinder. The second axis of the cylinder corresponds to its radial direction.

[0214] ALGORITHM 3InputOfze, 0feather, NOutput activationFunction CylinderActivation (dij, Oflze, QfeatherN)I 2r <_JZ"= 1K>] |di,7[0]- Olsize[0]|0{eather[0]r - Of£ze[l] |b ^ l.0 - J0{eather[1]if (df.7[0] < Ofze[0] ) and(r < Ofze[l])return 1.0if (d [0] > Ofze[0] + oeather[0] ) and (r > Ofze[l] + oeather[1]) return 0.0retum min(max(a, 0), max(b, 0)

[0215] An example activation function for a N-dimensional bicone can be written as follows:

[0216] ALGORITHM 4Input dt, Ofize, o[eather, NOutput activationFunction BiconeActivation (dj7, Oflze, QfeatherN)D24100WO01<jj,j w0?lze[n]size [1] * COS - / 70 tan-i(2&21)2Voflze[1]'H(cos0 —sin 01bLin 0 cos 0 -Iv <- [dt [0],rp <- R vifp[l] <= sreturn 1.0

[0217] The bicone activation function above is designed such that a linear falloff of the activation function occurs along the normal direction from the surface of the bicone.Sensory Objects Having Arbitrary Shapes

[0218] The foregoing discussion has provided examples of sensory objects having relatively simple shapes, including but not limited to spheres, ellipsoids, cylinders, rectangular prisms and bicones. Some disclosed examples involve sensory objects having more complex shapes, which may be defined in various ways. Some such sensory objects may be said to have “arbitrary” shapes. Such arbitrary shapes could be any shape that a content creator chooses.

[0219] Figures 9A, 9B and 9C show examples of the playback environment of Figure 8A in which a sensory object is defined by a set of vertices. As with other figures provided herein, the types, number and arrangement of elements shown in Figures 9A-9C are merely provided by way of example. Other implementations may include more, fewer and / or different types, numbers and / or arrangements of elements.

[0220] According to these examples, Figures 9A-9C include the following elements:800: A playback environment, which in this example is a room in which a lightscape experience is being rendered;801: a light fixture in the room 800;D24100WO01902: a sensory object, which is a light object defined by a set of vertices in this example;907: a centroid 807 of the light object 902; and810: a vector between a centroid 807 of the light object 802 and the light fixture 801.

[0221] Figure 9A also shows vertices 913a, 913b, 913c, 913d, 913c, 913f and 913g, which may be used to define the light object 902. The light object 902 is an example of an “arbitrarily-defined” sensory object. In some examples, the light object 902 of Figure 9A may be a 3D light object that includes additional vertices that are not shown in Figure 9A. An arbitrarily-defined sensory object shape that is defined by a set of vertices may be referred to herein as a “mesh.”

[0222] Figures 9B and 9C show two examples of feathering regions that surround the light object 902. In the example shown in Figure 9B, the feathering region 917a can be defined by a normal vector to the mesh 915 — such as the normal vector 912 — at the location that the activation is being evaluated. In such instances, the activation may be determined by (a) computing the distance of the location of interest along this normal vector and (b) applying whatever activation function has been defined in the feathering region — e.g., defined by the content creator — to determine the activation value. An activation function that is defined in a feathering region also may be referred to herein as a feathering function. In one example, the activation function may be defined by a linear falloff along the normal vector 912 from an activation value of 100% or 1 at the surface of the mesh 915 to an activation value of 0% or 0 at the outer surface of the feathering region 917a. In other examples, the activation function may be defined by a non-linear decay function along the normal vector 912 from an activation value of 100% at the surface of the mesh 915 to an activation value of 0% at the outer surface of the feathering region 917a.

[0223] In the example shown in Figure 9C, the feathering region 917b is a region between the mesh 915 and a mesh 925 that surrounds the mesh 915. In this example, the mesh 925 is defined by the vertices 923 a, 923b, 923c, 923d, 923e and 923f. The feathering function in the feathering region 917b may be defined in a variety of ways, such as by a gradient that is defined along normals to the surface of the mesh 915, by a gradient that is defined along normals to the surface of the mesh 925, etc.D24100WO01Extended Light Fixtures

[0224] Light fixtures are not literally point sources of light in a playback environment. Some light fixtures may be referred to herein as extended light fixtures, or simply as extended lights. Extended lights, when actuated, may cause one or more areas or regions of a playback environment to be noticeably illuminated. From a viewer’s perspective, the light source is effectively extended across such areas / regions.

[0225] Figure 10 shows examples of extended light fixtures in a playback environment. In this example, the playback environment 1000 is a room that includes extended light fixtures 1005a and 1005b. As with other figures provided herein, the types, number and arrangement of elements shown in Figure 10 are merely provided by way of example. Other implementations may include more, fewer and / or different types, numbers and / or arrangements of elements.

[0226] According to the example shown in Figure 10, the extended light fixture 1005a is mounted on the ceiling 1010 and illuminates a volume that includes portions of the ceiling 1010 extends to the wall portion 1020a and the sofa 1015. In this example, the extended light fixture 1005b is positioned on the floor behind an arm of the sofa 1015 and illuminates a volume that includes the wall portion 1020b — mainly below the windows 1025 — and portions of the ceiling 1010.

[0227] The activation of extended light sources may be defined according to extended light activation volume data. The extended light activation volume data may indicate one or more luminance volumes of a playback environment in which a corresponding extended light fixture is estimated to cause luminance that is perceivable by a human viewer. The luminance volumes may, for example, be defined according to parametric or arbitrary shapes. The luminance volume(s) may include one or more volumes illuminated by direct light from the extended light fixture, one or more volumes illuminated by reflected light from the extended light fixture, or combinations thereof.

[0228] The light activation caused by the interaction of a light object with an extended light fixture may be determined in various ways. Some methods of determining the light activation caused by the interaction of a light object with an extended light fixture may involve computing the inner product of the extended light fixture and the light object in some activation space. This process may involve projecting both the extended light fixture and the light object into some space in which the inner product can be computed. The spaceD24100WO01may, for example, be a spherical surface, a cube surface or a volume. The space may be either egocentric or allocentric.

[0229] Computing activation for an extended light fixture according to inner-product-based methods may potentially be based on an accurate computation of the intersection of the extended light fixture and the light object. In some examples, determining the light activation according to an inner-product-based method may involve some perception weighting when projecting onto the space in which the inner product is computed.

[0230] However, extended lights by their nature do not lend themselves to high-fidelity spatial resolution, in part because extended lights may illuminate large areas of space within the room to varying degrees. Moreover, the extended light activation volume data will generally be a mere approximation of the luminance volume(s) of the playback environment in which a corresponding extended light fixture causes luminance that is perceivable by a human viewer. In addition, computing inner products across a space which represents a playback environment, even at a low resolution, can be extremely computationally complex.

[0231] Accordingly, some disclosed methods provide a relatively simpler process of calculating the activation caused by the interaction of a light object with an extended light fixture. Some such methods involve a trade-off of the accuracy of inner-product-based methods for a process that is more convenient and / or computationally less complex. Some such methods involve projecting a set of test light object coordinates onto an extended light activation volume and evaluating activations of a corresponding set of projected test light object coordinates.

[0232] Figures 11 A, 1 IB and 11C show examples of projecting light object coordinates onto extended light activation volumes. As with other figures provided herein, the types, number and arrangement of elements shown in Figures 11 A-l 1C are merely provided by way of example. Other implementations may include more, fewer and / or different types, numbers and / or arrangements of elements.

[0233] Figures 11A-11C share the following elements:1150: a playback environment, which in this example is a room in which a light object and an extended light fixture are located; and1151A, 1151B, 1151C, 1151D, and 1151E: test light object coordinates; and 1152A, 1152B, 1152C, 1152D, and 1152E: projection points, which are projections of the test light object coordinates 1151A, 1151B, 1151C, 1151D, and 1151E, respectively, onto a surface of an extended light activation volume. Projection points also may be referred to herein as “projected test light object coordinates” or “projected points.”D24100WO01

[0234] In Figure 11 A, a light object 1122 A having the shape of a rectangular prism — also referred to herein as a cuboid — is shown. In this example, test light object coordinate 1151 A corresponds with a centroid of the light object 1122A, whereas test light object coordinates 1151B, 1151C, 1151D and 1151E are vertices of the light object 1122A. One way to project test light object coordinates onto the extended light activation volume involves determining the corresponding points on the surface bounding the extended light volume. According to this example, the test light object coordinates 1151 A, 1151B, 1151 C, 1151D, and 1151E have been projected onto projection points 1152A, 1152B, 1152C, 1152D, and 1152E, respectively, of a surface 1154A of an extended light activation volume 1155 A. More generally, K test light object coordinates produce K projection points f-j.

[0235] In some examples, each of the projection points 1152A-1152E may be evaluated according to an activation function of the light object 1122A. In some such examples, the maximum resulting activation value may be selected and may be used to activate a light fixture corresponding to the extended light activation volume 1155A.

[0236] In the case of a cuboid, as shown in Figure HA, the extended light volume has the following properties:L?ose [Rwis the position of the extended light 1155 A in the room 1050 reference frame;Ljslzeeis the size of the extended light 1155 A in the room 1050 reference frame; andRL6x Nis the rotation matrix representing the extended light 1155 A with respect to the room 1050 reference frame.

[0237] In some examples, a method may involve computing the distance vector, e- j Eusing the reference frame of the room 1150 and the extended light 1155A, between the center of extended light position lJ’0Sand any point Vkwe wish to project derived from the geometry of the extended light 1155A, e.g., as follows:

[0238] The distance vector, e- j may then be transformed into the light object 1122A reference frame, e.g., as follows:eij =RLei',jD24100WO01

[0239] In the foregoing expression, represents a rotation matrix that indicates the orientation of the light object 1122A with respect to the room 1150 reference frame. In some examples, element of e^j may be clipped to the surface 1154A of an extended light activation volume 1155A, e.g., as follows:f^ = clip(eiJ-qize, I^ize)that is, / [n] = sign( / [n]) mm( abs( / [n]), Ljfze)

[0240] In some examples, may then be transformed back into the room reference frame to use as the point to evaluate a light object activation function, e.g., as follows:d =Rikj

[0241] The test light object coordinates V to project and test can be derived from one or more of the following:• The shape of the light object;• The shape of the extended light;• The relative position / distance of the light object and the extended light;• The relative orientation of the light object and the extended light; and / or• The mode of the flexible lightscape Tenderer (FLR).

[0242] For example, if the distance between the extended light and a light object is close to the sum of their sizes, then edge effects or aliasing-type artifacts may occur. In such cases, it may be desirable to project additional points to increase the accuracy of this approach.

[0243] Some methods involve choosing test light object coordinates only based on the shape of the light object. The following examples are illustrative:• For a cuboid, project the centroid and bounding vertices (which also happen to be the vertices of the cuboid itself);• For a bicone, project the centroid and the vertices of a cross-section through the bicone; and• For cylinders and ellipsoids, project their centroid and bounding box vertices, some of which may not coincide with points on the light object’s surface, but which are suitable for determining points on the extended light surface to test the light object activation function. Figure 11C shows one such example.D24100WO01

[0244] Extended light volumes may be parameterised as cuboids, as shown above, as ellipsoids, cylinders or any other parameterizable volume. To use these volumes, we need only define a projection mechanism for the test light object coordinates V-' derived from the light-object onto the extended light volume surface. The test light object coordinates need not be perfectly accurate so long as they are somewhat continuous in order to reduce temporal artifacts when light-objects move throughout the room.

[0245] Figure 1 IB shows an extended light activation volume 1155B that is defined by an arbitrary mesh. In this example, the projected points(1152A-1152E) correspond to the points on the surface 1154B of the extended light activation volume 1155B that are closest to the centroid (1151 A) and vertices (1151B-115 IE) V- of the light object 1122A.

[0246] Figure 11C shows the projection of the vertices of the bounding box (1151B-115 IE) of an ellipsoidal light object 1122B onto the surface of the extended light cuboid volume (400). Because the test light object coordinates that are being projected onto the surface 1154C of the extended light activation volume 1155C do not actually coincide with the geometry of the light object 1122B, the control system implementing this method should ensure that it does not consider intersections of the test light object coordinates with the extended light activation volume 1155C as activations. Instead, the test light object coordinate should be projected onto the 1154C of the extended light activation volume 1155C and then evaluated using a light object activation function. One such example is test light object coordinate 1151C, which is located within the extended light activation volume 1155C, but which is projected to projection point 1152C on the surface 1154C.

[0247] Accordingly, instead of computing the activation for an extended light fixture according to an inner-product-based method or the equivalent, some alternative and simpler methods of estimating the activation involve determining which points in space to evaluate for a light object activation function, whether the light object is parametrically defined or is an arbitrary light object. Such projection-based methods are generally simpler and less computationally intensive than inner-product-based methods. Therefore, projection-based methods may potentially save time, reduce power consumption, or both.

[0248] Figure 12 is a flow diagram that outlines one example of a method that may be performed by an apparatus or system such as those disclosed herein. The blocks of method 1200, like other methods described herein, are not necessarily performed in the order indicated. In some implementation, one or more of the blocks of method 1200 may be performed concurrently. Moreover, some implementations of method 1200 may includeD24100WO01more or fewer blocks than shown and / or described. The blocks of method 1200 may be performed by one or more devices, which may be (or may include) one or more instances of control system such as the control system 110 that is shown in Figure 1 A and described above. For example, at least some aspects of method 1200 may be performed by an instance of the control system 110 that is configured to implement the multi-sensory rcndcrcr of Figure 3. Some aspects of method 1200 may be performed by an instance of the control system 110 that is configured to implement the lightscape Tenderer 501 of Figure 5.

[0249] In this example, method 1200 involves controlling a set of one or more controllable actuators of a playback environment. Here, block 1205 involves obtaining, by a control system, actuator data for the set of one or more controllable actuators. In some examples, the set of one or more controllable actuators may include one or more light fixtures, one or more haptic devices, one or more air flow control devices, or combinations thereof.

[0250] According to this example, block 1210 involves receiving, by the control system, object-based sensory data and sensory object geometry metadata, the object-based sensory data including a set of one or more sensory objects and the sensory object geometry metadata indicating a sensory object shape and a sensory object size of a corresponding sensory object of the set of one or more sensory objects. In some instances, the object -based sensory data may include other types of sensory object metadata.

[0251] In this example, block 1215 involves rendering, by a sensory data Tenderer implemented via the control system, the object-based sensory data to produce one or more actuator control signals. According to this example, the rendering is based at least in part on the actuator data and the sensory object geometry metadata.

[0252] According to this example, block 1220 involves providing, by the control system, the one or more actuator control signals to one or more controllable actuators of the set of controllable actuators.

[0253] In some examples, the sensory object shape may be a three-dimensional (3D) shape. According to some examples, the 3D shape may be a non-spherical shape, such as a polyhedron shape, an ellipsoidal shape, a cylindrical shape or a bicone shape. In some examples, the 3D shape may be defined by a set of vertices.

[0254] According to some examples, the sensory object geometry metadata may include a rotation parameter, which may be a 3D rotation parameter.

[0255] In some examples, the sensory object geometry metadata may include feathering information. The feathering information may define a feathering region in which an actuator activation value is between zero and one, wherein one is a maximum value. The featheringD24100WO01information may include a feathering function to be applied in one dimension, two dimensions or three dimensions. The feathering information may be based, at least in part, on a normal to one or more sensory object surfaces, on a normal to one or more surfaces that define a feathering volume, or combinations thereof. In some examples, the sensory object geometry metadata may include a two-dimensional shape or a three-dimensional shape within which the actuator activation value is one. The two-dimensional shape or the three-dimensional shape may reside within the feathering region.

[0256] According to some examples, the rendering may be based, at least in part, on a precomputed data structure. In some examples, the data structure may be an acceleration data structure.

[0257] In some examples, the set of one or more controllable actuators may include a set of one or more light fixtures, and the object-based sensory data may include light object data and light object geometry metadata. In some such examples, the sensory data Tenderer may be, or may include, a lightscape renderer configured to produce control signals for the set of one or more light fixtures based at least in part on the light object geometry metadata and the actuator data.

[0258] According to some examples, the actuator data may include extended light activation volume data for one or more extended light fixtures. In some examples, the extended light activation volume data may indicate one or more luminance volumes of the playback environment. Each of the one or more luminance volumes may be a volume of the playback environment in which a corresponding extended light fixture is estimated to cause luminance that is perceivable by a human viewer. According to some examples, the one or more luminance volumes may include one or more volumes illuminated by direct light from the extended light fixture, one or more volumes illuminated by reflected light from the extended light fixture, or combinations thereof.

[0259] In some examples, the extended light activation volume data may include one or more rotation parameters corresponding to rotation of the corresponding extended light fixture. According to some examples, the rendering may involve determining an activation of an extended light fixture by computing an inner product of extended light activation volume data and light object geometry metadata in an activation space.

[0260] According to some examples, the light object geometry metadata may include a set of light object coordinates. In some such examples, the rendering may involve determining an activation of an extended light fixture by projecting at least one object coordinate of the set of light object coordinates onto an extended light activation volume indicated by theD24100WO01extended light activation volume data. In some such examples, the rendering may involve determining the activation of the extended light fixture by projecting a set of test light object coordinates of the set of light object coordinates onto the extended light activation volume and evaluating activations of a corresponding set of projected test light object coordinates.Rendering Snapped Spatial Objects

[0261] Due to the nature of the spatial activation functions in the Tenderer, it is possible that some spatial sensory objects do not activate any of the actuators in the endpoint. This can be undesirable for some content creators whose intent is for a spatial sensory object to be rendered in the endpoint even if spatial fidelity is decreased. This intent can be conveyed in metadata associated with a spatial sensory object, e.g., by snapping metadata — also referred to herein as snap metadata — indicating that a corresponding spatial sensory object should be rendered even if spatial fidelity is decreased. The snap metadata may, for example, be implemented by a flag or a bit that is set or not set.

[0262] In some examples, the Tenderer may augment the spatial actuator activation matrix (AAM) if a spatial sensory object has its snap metadata flag set, such that the Tenderer causes one or more actuator control signals to be generated for the corresponding spatial sensory object even if no actuator control signal would otherwise have been generated. The computation of the AAM for a spatially activated object may be preceded by the computation of the offset tensor, d G Rw°x Na x 3which is the tensor composed of the offset vector, dtj, between all objects and actuators in the light object reference frame. The offset vector dLis defined above, as follows:d:> J j = R1d'iL> J;

[0263] As noted above, d j Grepresents the offset vector in the playback environment reference frame and may be determined as follows:d'. = i?os- Ofos

[0264] In the foregoing expression, O?osrepresents the position of the Ithsensory object and L^osrepresents the position of the jthactuator. The distance matrix D G Rw°x Na, which is defined in the arbitrary spatial shape activation section above, may be computed by taking the L-2 norm across the N index of the d tensor. If the spatial object activation function is isotropic then it will take Dtj as an argument, otherwise it will take dtj.D24100WO01

[0265] Some activation functions may be non-linear. For example, isotropic activation functions which consume dt- may be spherical. Non-isotropic activation functions may be cuboid, bicone or ellipsoid or spherical (when elements of the size vector are not all the same which means the activation function is spherical by default).

[0266] In some instances, if the distance between the sensory object and the actuator is too great, the actuation value provided by some activation functions may be 0. Fundamentally, this is how sensory objects can fail to be rendered in an endpoint. In order to avoid this, some methods of rendering snapped spatial objects may involve one or more of the following:• Scaling Dtj or the elements of dt to reduce their magnitudes;• Augmenting the spatial activation function(s) to account for an additional activation margin.

[0267] In some examples, the control system may render snapped spatial objects with the addition of metadata that is computed ahead of time, e.g., during a Tenderer configuration process and / or a Tenderer tuning process. If the Dtj or the elements of di:j are scaled, then the Tenderer can render snapped spatial objects with knowledge about a snapped spatial object’s potential activation across all light fixtures. Rendering a snapped spatial object can be done with information regarding the spatial object’s activation function (e.g., the distance at which the spatial object engages with an actuator). Some examples may involve optimizing the rendering of a snapped spatial object according to the effect of the snapped spatial object on multiple (e.g., all) actuators. Some such examples may involve providing a scaling of activation that is coherent across all of the actuators in a playback environment. In one such example, each of the elements of DL 7or dtmay be scaled in a way that is equivalent to augmenting the position vector of the snapped spatial object.

[0268] In some instances, the content creator may want a spatial object to always be present in the final actuation vector. This functionality may be implemented by setting (snapping) the spatial object’s position to be equal to that of the nearest actuator, e.g., as follows:f^pOS _ jPOSUL ~Lk

[0269] In the foregoing expression, k is the index of the spatial object with minimal scalar distance and may be expressed as follows:k = argmirij ( i pD24100WO01

[0270] Some examples also may involve applying a scaling to the jthcolumn of j or Dtj according to precomputed metadata. In some examples, this scaling may be computed ahead of time to ensure that preferred or undesired spatial directions, particular actuators, or both, may be enhanced or penalised accordingly. In some instances, the degree of enhancement or penalization may be a function of the type of the spatial object, of properties of a particular actuator, etc. Moreover, in some examples the argmin() can be completely replaced by using a look up table (LUT) or other data structure (DS), such as an an acceleration data structure, e.g. a binary tree, the values of which may in some examples be computed at Tenderer configuration / tuning / initialisation time. That is, we can omit the expensive computation of Dij or di:j by referring to a DS, e.g., as follows:k = DS 0?os)

[0271] Since we are using the DS to return an index, it is not desirable for any interpolation to occur. In some examples, the DS may be computationally cheap to prepare and evaluate. In some such examples, the DS may simply be a LUT or an array expressed as follows:k = DS xt ^Zi]

[0272] In some examples, the array may be indexed as follows:Xi =~dTpos

[0273] In the foregoing expressions, [■] is the floor operator and dx, dy and dz are the spatial resolution of the (x,y,z) dimensions of the DS, respectively. Care should be taken to ensure that the DS is indexed such that (%f, yt, ~zt) are not out of bounds. This can be done either by clipping the indices so they are within the size of the DS or clipping the spatial object position to the normalised coordinate space of the endpoint. After the index k has been looked up, in some examples the Tenderer may simply set the k!,icolumn of the Ithrow of A to be 1, effectively snapping the spatial object to that index.

[0274] However, in some instances the Tenderer may be snapping the spatial object to a small actuator in the middle of an array of actuators. In such instances, simply setting A[ j =D24100WO011 would generally provide undesirable actuations. To mitigate this effect, in some examples the Tenderer may set the spatial object’s position to the position of the kth actuator, as follows:^pos _ jPOSui ~Lk

[0275] The Tenderer may then re-evaluate the object- actuator activation function across the endpoint.

[0276] In some alternative examples, the DS may return multiple actuator indexes, e.g., as follows:K= DS[xt

[0277] In such instances the Tenderer may set all columns corresponding to indices of the K vector of the i‘hrow of A to be 1, effectively snapping the spatial object to multiple actuators at once. This is potentially useful, because the configuration generator is not bound by computational complexity that is practical at typical frame rates of the Tenderer. As a result, the configuration generator can determine an optimal K vector for a given object position, and potentially also one or more of the following:• Object size;• Object velocity;• Object brightness;• Object color.Where the DS would also be indexed by these additional parameters.

[0278] The K vector may, in some instances, return the indices of optical actuators in the order of decreasing luminance. In some examples, a spatial object may include metadata that indicates the total amount of brightness associated with the spatial object. In such instances, the Tenderer may traverse through the K vector, setting the columns ofto be 1 until the cumulative luminance associated with that spatial object is sufficient according to the metadata.

[0279] Some implementations may involve augmenting the snapping of spatial objects by first computing A and then, for all spatial objects which are required to snap to actuators and which do not have at least one actuation value of 1 (in other words, there is at least a single 1 in the objects row of A ), performing the snapping process after analyzing the already-computed A matrix. Some such examples may involve applying a scaling to the jthcolumn of di or Di when computing the argmin() above to find k. The scaling may be desirable inD24100WO01order to reduce the impact of snapping to actuators already activated by other objects that may be:• Higher-priority or on a higher layer, which could cause the snapping of the current spatial object to be useless if the blending and mixing laws would not result in the current spatial object being actuated; or• Lower priority, which could cause causing distortion to the already-rendered scene.Rendering Volumetric Fields

[0280] The term “volumetric field” as used herein refers so a field within a volume, the field being defined by functions. The functions may, for example, be parameterizable by a content creator. Volumetric fields may or may not be spatially uniform, depending on the particular implementation. In some examples, a content creator may specify in the content metadata whether a volumetric field is spatially uniform. Functions which may be parameterizable across the volumetric field include, but are not limited to, the following:• Spatially modulated basis functions, such as sinusoidal functions that are a function of the spatial position across the field;• Temporally modulated basis functions, such as sinusoidal functions that are a function of time;• Signal mixing functions;• Signal multiplication functions;• Activation functions, such as:o Non-linear transformations e.g. a = x2; oro Non-linear thresholding, e.g., a = min(x, b); and / or• Deterministic generators parameterised across the volume, e.g., pseudo-random noise.

[0281] The volumetric field may be sampled at spatial intervals, which may be either uniform — that is, sampled at regular intervals across the space of the endpoint — or non-uniform. Non-uniform sampling may, for example, be based on information in the content metadata. In one such example, information in the content metadata may indicate, e.g., sampling centred at a display screen, such as a television (TV) display screen. In some examples, the sampling may be log-space sampling. According to some examples, the volumetric field may be sampled temporally each time the Tenderer runs a process call.

[0282] Volumetric fields, volumetric textures, or both, can be used to augment and / or modulate the properties of spatial objects across an endpoint. For example, a volumetricD24100WO01field may be created in which the amplitude is spatially and temporally modulated. This amplitude may then be applied to any spatial objects in the endpoint and only then will the spatial and temporal modulations affect the actual actuators. One example may involve “flicker” modulating of all light objects in a room.

[0283] The output of a volumetric field can be a scalar or a vector. In some examples, the output of a volumetric field may be used to directly affect an actuator. For example, a volumetric field may produce an RGB color tuple, which may be directly applied to one or more actuators. In some examples, the output of a volumetric field may be used to modulate one or more actuators. For example, a scalar volumetric field may modulate the amplitude of the actuators in a playback environment.

[0284] According to some examples, the Tenderer may generate a parametric volumetric field tensor VPG IRWxX Nyx NzX NP, where Nx, Nyand Nzrepresent the numbers of spatial samples in the x, y and z axes of the room. These numbers are equal when the spatial sampling is uniform and the room is isotropic. Nprepresents the dimensionality of the volumetric field. In this example, Np= 1 if the volumetric field is a scalar field and Np> 1 if the volumetric field is a vector field.

[0285] If the volumetric field is a scalar field and will modulate the amplitude of the actuators in the endpoint, then the volumetric field may be implemented as follows:A' = A Q Vp

[0286] In the foregoing expression, A represents an actuator activation matrix (AAM), such as the light activation matrix (LAM) 740 that is described with reference to Figure 7, 0 represents Hadamard multiplication and A ’ represents the modified AAM resulting from the implementation of the volumetric field.

[0287] One way to realize a volumetric field is to specify (a) a spatial modulation function parameterised by the volumetric field’s origin and the density of the volumetric field and (b) a temporal modulation function. In some examples, the temporal modulation function may be parameterised by a time-frequency relationship and a thresholding function. One such volumetric field that outputs a vector may be expressed as follows:< > = mod((l)x+ (f)y+ <pz, 2n)In the foregoing expression, mod(a, b) represents a modulo function which returns the remainder of a divided by b, and < >x, < >y, < >zrepresent the spatial phases associated with eachD24100WO01of the dimensions of a playback environment at the position (x,y,z). In some examples, < >x, < >y, < >zcan be computed as follows:= (* “ Ofos)fx+ cf>tfy= (y - C»£os) / y + <t>t< Pz = ( / - Or)fz+ <t>t

[0288] In the foregoing expression, fx,fy,fzrepresent the spatial frequencies associated with the x,y,z dimensions, respectively and < >trepresents the phase of the temporal modulator, which may be computed as follows:<ft = t ft

[0289] In the foregoing expression, t represents the current time and ftrepresents a timefrequency temporal modulation parameter. The corresponding volumetric field tensor VPmay be expressed as follows:> r c if ( / ) < ap.^x.y.z) otherwise

[0290] In the foregoing expression, a represents a threshold between zero and 2n and is a parameter that a content creator may control, c represents a codeword vector and Os represents a vector of zeros having the same length as c. For a lightscape modality, c could be an (R, G, B) codeword, for example.

[0291] According to some examples, c itself could be generated in various ways, such as:• as a function of spatial position;• as a function of some other parameter such as time, or of a parameter interactively set by a user;- by a random generator function, etc.

[0292] A content creator may choose to have some spatial dimensions not dependent on the origin of a volumetric field. For example, one could set < >zas follows:<fz = O - 0.5)z+ (f)t

[0293] In order to compute Vp,(X,y,z)onemay choose to substitute the positions of the actuators for the (x,y,z) locations. Alternatively, one may choose to only evaluate the volumetric field at a grid of spatial locations that arc independent of the actuator locations and then perform some interpolation or activation.D24100WO01

[0294] Some examples may involve using precomputed data obtained at tenderer configuration / tuning / initialization time to bound or scale the parameterized functions used to realize a volumetric field. For example, one may use such precomputed data to limit the range of amplitude modulation according to knowledge of the actuator capabilities, to scale the spatial-frequencies according to an endpoint room scale, etc.

[0295] Some examples may involve limiting the range of an amplitude-modulated scalar volumetric field that may simply be a temporally modulated field, for example as follows:^p,(x,y,z) = 1 -

[0296] Some examples may involve augmenting the foregoing expression as follows:^P,(x,y,z) 1

[0297] In the foregoing expression, a represents a modulation depth limiting factor that will avoid actuators being continually driven to actuation amplitudes below a when one sets A' = A 0 VP. There are many ways in which one can augment the modulation depth, for example as follows:vP,(x,y,z) = 1 - a (1 +The foregoing expression leaves the modulation result balanced.

[0298] A volumetric field can provide various types of functionality, including but not limited to the following:• Augmenting the values of A directly if the field is scalar as shown earlier;• Augmenting the object-actuator activation functions directly or indirectly, for example:o By modulating the distance matrices, e.g. Etj = Di Vp for Np= 3 volumetric fields or E;j = di VPfor scalar volumetric fields;o By modulating spatial object sizes;o By modulating spatial object positions;o By modulating actuator positions;• Augmenting the codewords, e.g., for lightscapes the R channel of an RGB codeword may be modulated by a scalar volumetric field; and / or• Producing a vector field of codewords which will be mixed in with the remainder of the objects being rendered.D24100WO01

[0299] If a vector volumetric field produces codewords and activation values for each actuator in the endpoint, then we have Avwhich is a Nf by Namatrix in which Nfrepresents the number of volumetric fields being rendered and each row of Avrepresents the volumetric field activation for a particular field for every volumetric field being rendered (there may be multiple volumetric fields in the content stream being rendered). Then, we can mix the volumetric field in byRendering Volumetric Textures

[0300] Volumetric textures are similar to volumetric fields in the sense that they provide either a scalar or vector field over a spatial coordinate domain of the endpoint. The main difference is that the field corresponding to a volumetric texture is derived from data, not by functional parameters provided by the creator. For example, a content creator may choose to paint a 2D or 3D image and use this as the basis for the volumetric field. A greyscale image would produce a scalar volumetric field. The content creator can take this asset and determine how it is sampled to produce the volumetric texture. If the image is 2D, for example, the content creator may choose to place this asset on the XY plane (parallel to the floor of the playback environment) and may choose to make the field constant along the Z axis.

[0301] Volumetric textures may be combined with volumetric fields to create desired effects. For example, a scalar volumetric texture may be used to define some arbitrary function over the spatial domain which drives time-frequency oscillators at particular spatial locations.

[0302] Volumetric textures can be used in the same way that volumetric fields are used to augment the actuator activation matrix and object activation functions, e.g., as described in the previous section.

[0303] Because volumetric textures are not defined in terms of parameterizable functions, it is not possible to evaluate them at arbitrary (x,y,z) spatial locations. Thus, we cannot evaluate them at the actuator positions, and we are only able to perform some sort of interpolation or resampling of the volumetric texture if we want to directly project it onto actuators to compute Avor Vp. Evaluating the volumetric texture at arbitrary (x,y,z) spatial locations to compute Avor Vp. may, for example, involve:Nearest-neighbour sampling;Trilinear interpolation; orD24100WO01• Tricubic interpolation.

[0304] If the actuator positions are fixed in the endpoint, precomputed coefficients or other values may be used to speed up the interpolation process. If the volumetric texture is static, the interpolated values may be precomputed once and stored for subsequent use.Interpolated values also may be precomputed once and stored for subsequent use if the volumetric texture is not static, but periodic and the total amount of data is not too large to store.Rendering Images and Videos

[0305] Images and videos differ in nature to volumetric textures (and fields) in the sense that images and videos are generally not evaluated or sampled across the spatial domain of an endpoint. Instead, images and videos are generally rendered to one or more actuators comprising an array of actuators that provide sufficient resolution and coverage to render an image or video. The reader will note that, as used herein, the terms “images” and “videos” are not necessarily optical images and videos. For example, an image may simply be a 2D or 3D signal with regular uniform spatial sampling. One example of a non-optical image is a haptic signal that is defined across an array of spatial points that could, for example, be rendered to an array of actuators on a vest. A video is simply an image in which the signal changes over time.

[0306] A content creator may import an image or video into their sensory creation tooling and associate the image or video with a sensory object. In some instances, a content creator may create a sensory object corresponding with an image or video. The position and size of this sensory object can determine the bounding region in which the sensory object is able to be rendered upon a suitable array of actuators.

[0307] A content creator may also determine the way in which an image or video is rendered upon an array of actuators, for example by defining one or more of the following in content metadata corresponding to the image or video:• The method of resampling / resizing the image or video to an actuator array having a different aspect ratio than that of the asset;• Minimum capabilities of the actuator array required to render the image or video, including but not limited to:o The number of elements in the actuator arrayo fhe maximum spatial distance between elements of the actuator array; o The maximum spatial distance between actuator sub-arrays; and / orD24100WO01o The actuation capacity of the actuator array, such as the color gamut for light fixtures or the frequency range for haptic actuators.

[0308] Figure 13 shows an example of an image asset that is associated with a sensory object. According to this example, the image asset 1300 is a multichannel image.According to this example, the number of channels, Nc, =3. Here, the three channels are represented in Figure 13 by the image asset regions 1305, 1310 and 1315. In some examples, the image asset 1300 can be visualised using an RGB image rendered according to the content creator’s specifications. In such examples, the colors shown in the regions 1305, 1310 and 1315 may correspond to the colors that are intended by the content creator to be rendered in the playback environment. However, the image asset 1300 is not necessarily of the optical modality.

[0309] As with other figures provided herein, the types and numbers of elements shown in Figure 13 are merely provided by way of examples. Other implementations may include more, fewer and / or different types and / or numbers of elements. In some instances, an image or video may be “greyscale,” e.g., having a single channel where Nc=1 or having multiple channels Nc> 1.

[0310] Figures 14 and 15 show examples of the image asset of Figure 13 being rendered in different playback environments. As with other figures provided herein, the types and numbers of elements shown in Figures 14 and 15 are merely provided by way of examples. Other implementations may include more, fewer and / or different types and / or numbers of elements.

[0311] Figure 14 depicts an endpoint 1400 in which two sub-arrays of actuators (sub-arrays 1401 and 1402) are used to render a sensory object corresponding to the image asset 1300 of Figure 13. In this example, the image asset regions 1305 and 1310 are being rendered to the actuator regions 1405 and 1410, respectively, of sub-array 1401 and the image asset region 1315 is being rendered to the actuator region 1415 of sub-array 1402. According to this example, the actuator region 1415 includes the entire sub-array 1402. According to some examples, in order for the multiple sub-array rendering shown in Figure 14 to occur, the following conditions must be met:• Sub-arrays 1401 and 1402 must both be within a region defined by the sensory obj ect’ s position and size;• The sensory object’s metadata must indicate that rendering across multiple subarrays is allowed / enabled; andD24100WO01• The sub-arrays must meet the minimum spatial and actuation capabilities defined in the sensory object’s metadata.

[0312] Figure 15 depicts an endpoint 1500 in which a single array of actuators (array 1 01) is used to render a sensory object corresponding to the image asset 1300 of Figure 13. In this example, the image asset regions 1305, 1310 and 1315 of the image asset 1300 are being rendered to the actuator regions 1505, 1510 and 1515, respectively, of array 1501. According to some examples, in order for the type of rendering shown in Figure 15 to occur, the following conditions must be met:• The array 1501 must be within the region defined by the sensory object’s position and size: and• The array 1501 must meet the minimum spatial and actuation capabilities defined in the sensory object’s metadata.

[0313] According to some examples, in order for an array or subarray of actuators to meet the minimum spatial and actuation capabilities (e.g., as defined in a sensory object’s metadata), any required resampling factor must be allowed. Depending on the resampling method, some artifacts may be expected. For example, the “nearest neighbor” method can result in artifacts similar to that shown in Figure 15, in which the shape of the actuator region 1505 differs from the shape of the image asset region 1305.

[0314] For some resampling methods, interpolation coefficients should be calculated in order to combine multiple pixels in the image asset to produce a pixel value to be rendered to an actuator array or sub-array. These coefficients may be computed ahead of the rendering time in order to save computational cost at run-time.Normalizing the Rendering of Sensory Objects to Playback Environments Having Varying Shapes and Capabilities

[0315] A content creator generally creates sensory objects in a creation tool environment using one or more types of sensory content creation tools. The creation tool environment, which also may be referred to herein as a reference environment, has reference environment dimensions. For simplicity, the reference environment may be a virtual rectangular prism, such as a virtual cube in which the reference environment dimensions are equal along the x, y and z axes of the reference environment. The actual playback environment in which the sensory content is rendered will generally not be a cube-shaped environment. In some instances, the playback environment may be a vehicle. Even if the playback environment is a room having the shape of a rectangular prism, the room will typically not be cube-shaped.D24100WO01

[0316] Accordingly, it is often the case that the reference environment shape and the playback environment shape are different. Therefore, unless some type of normalization process is applied, at least some sensory objects that are rendered in the playback environment may have different sizes than the content creator intended.

[0317] For example, as described above with reference to Figure 7, the lightscapc Tenderer 501 may be configured to compute a projection of light objects 705 created in a reference environment onto the light fixtures of a playback environment, to form the light activation matrix (LAM) 740. This may be done by computing the spatial interaction between light fixtures and light objects after projecting them onto the same spatial domain. If the reference environment shape and the playback environment shape are different, at least some light objects that are rendered by the lightscape Tenderer 501 may have different sizes than the content creator intended, unless some type of normalization process is applied.

[0318] Moreover, when sensory content — including but not limited to light-based sensory content — is created, the sensory content is often created to be congruent with a light object or effect being rendered on a video screen. It is important to control the projection process such that light objects and light fixtures attached to, or positioned close to, display screens in the endpoint retain spatial congruency.

[0319] This section describes methods for normalizing spatial object sizes according to the playback environment projection. Methods for normalizing the projection of light objects into a playback environment, while retaining creator intent associated with video screen congruency, are introduced. Some such methods involve normalization processes for playback environments having multiple display screens. Methods for normalizing light fixture activation levels based on physical density, for total brightness normalization, are also presented.

[0320] In this section, the origin of the playback environment coordinate system is defined as the bottom front left comer of the playback environment. For a room-type playback environment, the x axis is in the direction of the back wall of the room, the y axis is in the direction of the right wall and the z axis is in the direction of the ceiling.

[0321] Figures 16A and 17A show examples of light objects in a reference environment. Figures 16B and 17B show examples of the light objects of 16A and 17A and 18A projected into a playback environment. As with other figures provided herein, the types and numbers of elements shown in Figures 16A-17B are merely provided by way of examples. Other implementations may include more, fewer and / or different types and / or numbers of elements.D24100WO01

[0322] Figure 16A depicts a reference environment 10000 with reference to which content is created. According to this example, the reference environment 10000 is a unit cube. In this example, light objects 10010 and 10011 are within the reference environment 10000.

[0323] Figure 16B shows an example of a playback environment, which in this example is a room in which a lightscapc experience is to be rendered. In this example, the playback environment 1600 is a rectangular prism in which the depth (along the x axis ) is twice the width (along the y axis). According to this example, light objects 10010 and 10011 are rendered to areas 1601 and 1602 of lightstrips 10001 and 10002, respectively, in the playback environment 1600.

[0324] According to some examples, when the lightscape Tenderer 501 projects the light fixtures of the playback environment 1600 and light objects of the reference environment 10000 into a common space to compute the light activation matrix, the lightscape Tenderer 501 normalizes the coordinates of the light fixtures according to the size of the playback environment 1600. In the example shown in Figure 16B, the playback environment 1600 is twice as deep as it is wide and the lightstrips 10001 and 10002 run the extent of the room. Accordingly, in this example the lightstrips 10001 and 10002 map to normalized room coordinates on the range of [0, 1], As a result of this mapping, the light-objects 10010 and 10011, which were created in the reference environment 10000 having equal sizes, arc rendered in the playback environment 1600 with different sizes: light object 10011 appears to be twice as large as light object 10010 when rendered on lightstrips 10002 and 10001, respectively.

[0325] We can resolve this by configuring the lightscape Tenderer 501 to also normalize the light object sizes, Olze, by the same factor by which the light fixture positions are normalized. For example, normalizing the j‘hlight position can be expressed as follows:jposition r -ILP°sition[ n] = >.r7JV n E [0, N - 1]7 L J flsize[n]L J

[0326] In the foregoing expression, RslzeErepresents the N-dimensional vector describing the size of the endpoint (and is generally 3), [P°sltlonrepresents the nonnormalized position and P°sltlonrepresents the normalized position. If the lightscape Tenderer 501 is configured to normalize sensory object sizes by scaling them by the same amount that the sensory object positions are scaled in the above example, then the sizeD24100WO01distortion issue shown in Figure 16B can be avoided. For example, the lightscape tenderer 501 may be configured to scale the z'th sensory object’s size as follows:Ofize[nlOflze[n] =l.L JV n G [0, N - 111 L J7?slze[n]L J

[0327] In the foregoing expression, Q?lzerepresents the non-normalized sensory object’s size and Oflzerepresents the normalized sensory object’s size.

[0328] Figure 17A depicts a reference environment 10000 with reference to which content is created. As in the example of Figure 16 A, the reference environment 10000 is a unit cube. In this example, light objects 10020 and 10021 are within the reference environment 10000.

[0329] Figure 17B shows another example involving the playback environment of Figure 16B. According to this example, light objects 10020 and 10021 are rendered to areas 1701 and 1702 of lightstrips 10001 and 10002, respectively, in the playback environment 1600. In this example, the lightscape Tenderer 501 is configured to normalize sensory object sizes. Therefore, the relative sizes of light objects 10020 and 10021 are preserved: light object 10020 is the same size as light object 10021 when rendered on lightstrips 10002 and 10001, respectively.

[0330] However, for parametric and arbitrary sensory objects that are not isotropic, then in some examples the control system may be configured to determine the scaling factors as a function of the sensory object’s orientation in the playback environment, which can be represented as / ?, Gx N. To do so, the control system may first define an inverse scaling vector Slze[n], e.g., as follows:1S;slze[n] = - V n G [0, N - 1]1 L J / ?ilzc[n]L J

[0331] The control system may then transform this inverse scaling vector onto the sensory object’s axes, e.g., as follows:Qsize _ T size

[0332] The control system may then apply the scaling directly to determine the normalized sensory object’s size:Ofize[nlO1?lze[Ln]J=Q Jsl.ze, V n G [0, N - 1][n]L J

[0333] The control system may, in some examples, apply this type of scaling to the positions of any vertices defined as part of an arbitrary mesh.D24100WO01

[0334] The above-described types of normalisation may result in the relative position of sensory objects shifting. For example, two sensory objects created in a unit cube reference environment that are both 0.1 in size and separated by a distance of 0.1 will appear to be touching. When playback environment normalisation is enabled, these sensory objects may not be adjacent when rendered in the playback environment.Volumetric Field Normalisation

[0335] For volumetric fields in which spatial modulations occur, the control system may be configured to normalize the spatial frequencies of a corresponding sensory object such that the rendered scene retains the relative spatial densities in the content across all the dimensions in the playback environment. This may, for example, be accomplished by computing the following:fx = Pxfxfy = Pyfyfz = Pzfz

[0336] In the foregoing equations, fx, fyand fzrepresent the normalized spatial frequencies, fx, fyand fzrepresent the spatial frequencies associated with the volumetric field and px. py, pzrepresent scaling parameters. The scaling parameters may, for example, be determined by computing the relative scale of the dimensions of the playback environment. In some examples, the scaling parameters may be determined by computing the scale of the dimensions of a bounding box enclosing some or all of the actuators within the playback environment. In some such examples, one scaling parameter may be set to 1 and the other two dimensions may be computed relative to this scaling parameter of 1.Volumetric Texture Normalization

[0337] For volumetric textures, the control system may be configured to apply the scaling parameterspy, pzto the spatial coordinates of the endpoint when resampling the volumetric texture.Actuator Position Normalization

[0338] Rendering sensory objects to a screen domain is a generalization of a playback environment-scale normalization process. In order to flexibly render lightscape experiences, the Tenderer may be configured to map real-world coordinates of the actuators in theD24100WO01playback environment onto a normalized range that corresponds to the reference environment coordinate domain. In some examples, instead of applying an element-wise scaling using Rsize, a control system may construct a scaling matrix Ssize∈ ℝN x N, where Ssize= diag( then may perform normalization as follows:x̂ = Ssize· Position

[0339] In some examples, instead of using Rsizewhich describes the size of the entire playback environment, the control system may use Rsizethat describes a bounding region of the actuators of interest. For example, one bounding region may be a region defining a display screen domain. In this example, the actuators of interest would be actuators that will render content for that display screen domain. According to some examples, the control system may use a subset of the playback environment when scaling actuators to screen domains. In some examples, the control system may generalize the normalization process to also include one or more of the following:• A translation component, allowing the origin of the coordinate system to be shifted;• A rotation component, allowing the orientation of the playback environment to differ from that of the reference environment. This may be useful when anchoring the orientation of the playback environment, e.g., so that a TV screen is parallel to the plane corresponding to the front wall in the reference environment.

[0340] In some examples, a control system may be configured to combine rotation and translation into a generalised transformation matrix T ∈ ℝ(N+1) x (N+1), e.g., as follows:Tposition • positionLi - TLJ1 1Screen-Anchored Sensory Objects

[0341] The section above describes an actuator position normalization process in which the transformation matrix T is constructed according to the entire playback environment.Actuator positions can also be normalized to multiple domains. Some examples may involve a normalization process for screen domains. In some such examples, a region of the playback environment and a subset of the actuators within the playback environment may be projected to a normalized coordinate domain. Such methods can allow a content creator toD24100WO01create reference spatial content anchored to multiple screens in the reference environment, resulting in the Tenderer producing light content relative to multiple screens in the playback environment according to the reference spatial content. According to some examples, this process involves computing a transformation matrix T for each of two or more screen domains. In some examples, for every domain, the control system may use a spatially variant transformation matrix. Using a spatially variant transformation matrix allows the control system to produce piecewise transformations to support various playback environment configurations. Here, we use the term “spatially variant transformation matrix” to define a transformation matrix that changes discretely depending upon which subset of the playback environment to which the transformation matrix is being applied. Within that subset of the playback environment, the transformation matrix is not spatially variant in the traditional sense. In some alternative examples, a transformation matrix which is spatially variant may be used within each subset of the playback environment. Figure 19A — which is described in more detail below — shows an example of a reference environment that has two actuator domains, each of which is anchored to a different reference screen. These are examples of the “screen domains” referenced in this paragraph. Some examples may involve computing a transformation matrix T for each of these two screen domains. In some examples — such as when the screen domains include different- sized areas of the reference environment — for each screen domain, the control system may use a different transformation matrix.Actuator Array Domain Rendering

[0342] In some instances, sensory content may may be created that is similar in nature to screen-anchored sensory objects, in the sense that the spatial and temporal properties of the sensory objects may be designed relative to an actuator array, such as a light strip or a light grid. In one such example, a content creator may design a sensory effect to be produced by way of a spatial sensory object traversing over a spatial domain with a particular positional trajectory. The content creator may, for example according to sensory object metadata, convey their intent for this sensory effect to be rendered on a single actuator array, for example on a single light strip. In order to create actuator array domain effects, the content creator may specify, for example according to sensory object metadata, a global position to indicate where the sensory effect is to be rendered in the playback environment.

[0343] The normalisation of positions may, in some examples, be performed as described elsewhere herein regarding the normalization of screen- anchored objects. In someD24100WO01examples, there may be multiple actuators mapped into an actuator array domain.According to some such examples, these multiple actuators may be arrays themselves and may create a super-array.

[0344] Figure 18 A shows another example of a reference environment with reference to which content can be created. Figure 18B shows an example of an actuator domain for content creation in the reference environment of Figure 18 A. As with other figures provided herein, the types and numbers of elements shown in Figures 18A and 18B are merely provided by way of examples. Other implementations may include more, fewer and / or different types and numbers of elements.

[0345] According to this example, Figure 18A shows a “top down” view of the reference environment 10000, viewed along the z axis and perpendicular to the x and y axes. In this example, an actuator domain object 50001a, a reference screen 50005a and a reference person 50010 are within the reference environment 10000. In this example, Figure 18A shows the x and y extent of the actuator domain object 50001a within the reference environment 10000. Accordingly, the content creator has defined a reference region corresponding to the spatial extent of the actuator domain object 50001a within the reference environment 10000, intending that the actuator domain object 50001a will be presented in a corresponding playback region of a playback environment. According to this example, Figure 18A indicates a distance du from the reference screen 50005 to a reference person 50010.

[0346] In this example, Figure 18B shows an actuator domain 50000a within which a content creator may indicate the desired details of the actuator domain object 50001a of Figure 18A. In this instance, the content creator has placed actuator domain sub-objects 50002 and 50003 within the actuator domain 50000a. The actuator domain sub-objects 50002 and 50003 may, for example, be light objects at corresponding positions within the actuator domain 50000a at a snapshot in time corresponding to Figure 18B.

[0347] Figure 19A shows another example of a reference environment with reference to which content can be created. Figures 19B and 19C show examples of actuator domains for content creation in the reference environment of Figure 19A. As with other figures provided herein, the types and numbers of elements shown in Figures 19A-19C are merely provided by way of examples. Other implementations may include more, fewer and / or different types and numbers of elements.

[0348] According to this example, Figure 19A shows a “top down” view of the reference environment 19000, viewed along the z axis and perpendicular to the x and y axes. In thisD24100WO01example, the actuator domain object 50001b includes a reference screen 50005b and the actuator domain object 50001c includes a reference screen 50005c. Accordingly, the content creator has defined two reference regions corresponding to the spatial extents of the actuator domain objects 50001b and 50001c within the reference environment 19000, intending that the actuator domain objects 50001b and 50001c will be presented in two corresponding playback regions of a playback environment.

[0349] In this example, Figure 19B shows an actuator domain 50000b within which a content creator may indicate the desired details of the actuator domain object 50001b of Figure 19A. Likewise, Figure 19C shows an actuator domain 50000c within which a content creator may indicate the desired details of the actuator domain object 50001c of Figure 19 A. The examples shown in Figures 19A-19C are potentially advantageous for multi-screen playback environments, including but not limited to vehicle-type playback environments.Actuator-Density-Based Lux Normalization

[0350] Actuator-density-based lux normalisation refers to the process of scaling the brightness of the light actuators such that the lux (total emitted light per unit area) is consistent across the playback environment. For example, consider two equal-length lightstrips on opposing sides of the playback environment. In this example, both lightstrips have the same physical actuators (LEDs, drivers, circuitry, etc.), such that the same level of actuation of a single element on both lightstrips results in the same physical actuation in the playback environment. If one of the lightstrips has twice as many elements, then commanding both lightstrips to set all of their elements to the same codeword will result in the lightstrip with twice as many elements producing twice as much lux and being perceived by the user as being logarithmically brighter.

[0351] If the goal is to simply set endpoint-wide light setpoints to the actuators, then in some examples the control system — such as a control system implementing the lightscape Tenderer 501 of Figure 5 or Figure 7 — may be configured to scale the brightness of the jthactuator by a factor Bj computed at Tenderer configuration, tuning or initialisation time. The brightness of an actuator may be scaled as follows:out = min (1, Bj · Bi)D24100WO01

[0352] In the foregoing expression, the min() operator ensures that for actuators that have Bj > 1, we do not exceed the range of valid codewords. The brightness may be scaled using any suitable color model.

[0353] Moreover, the ithlight object may have a target lux value associated with it. In such cases, the control system cannot compute Bt— which represents the scaling for a particular object- actuator at configuration time — because this data will not be available at configuration time. In some such examples, the lightscape Tenderer will compute BL, potentially using Bj as a reference. The computation of j may be based, at least in part, on one or more of the following parameters:• The number of actuator elements the light object is being rendered across; and / or • The target lux value.

[0354] Given these parameters, the Tenderer may also augment the position of the light object — in other words, alter the light object / light fixture distance to make the light object seem closer to the light fixture — in order to ensure sufficient actuation (lux) is achieved.

[0355] Figure 20 is a flow diagram that outlines one example of a method that may be performed by an apparatus or system such as those disclosed herein. The blocks of method 2000, like other methods described herein, are not necessarily performed in the order indicated. In some implementation, one or more of the blocks of method 2000 may be performed concurrently. Moreover, some implementations of method 2000 may include more or fewer blocks than shown and / or described. The blocks of method 2000 may be performed by one or more devices, which may be (or may include) one or more instances of control system such as the control system 110 that is shown in Figure 1 A and described above. For example, at least some aspects of method 2000 may be performed by an instance of the control system 110 that is configured to implement the multi-sensory Tenderer of Figure 3. Some aspects of method 2000 may be performed by an instance of the control system 110 that is configured to implement the lightscape tenderer 501 of Figure 5 or Figure 7.

[0356] In this example, method 2000 involves controlling a set of one or more controllable actuators of a playback environment. Here, block 2005 involves obtaining, by a control system, actuator data for the set of one or more controllable actuators and playback environment dimension data indicating a playback environment shape. In some examples, the set of one or more controllable actuators may include one or more light fixtures, one or more haptic devices, one or more air flow control devices, or combinations thereof.D24100WO01

[0357] According to this example, block 2010 involves receiving, by the control system, object-based sensory data including a set of one or more sensory objects created in a reference environment, and reference environment dimension data indicating a reference environment shape.

[0358] In this example, block 2015 involves normalizing, by the control system, a size of one or more sensory objects by one or more factors based on the playback environment shape relative to the reference environment shape, to produce one or more normalized sensory objects.

[0359] According to this example, block 2020 involves rendering, by the control system, the one or more normalized sensory objects to produce one or more actuator control signals.

[0360] According to this example, block 2025 involves providing, by the control system, the one or more actuator control signals to one or more controllable actuators of the set of controllable actuators.

[0361] In some examples, the actuator data may include an actuator map. According to some such examples, rendering the one or more normalized sensory objects to produce one or more actuator control signals may involve projecting the set of one or more normalized sensory objects using the actuator map. In some such examples, projecting the set of one or more normalized sensory objects using the actuator map may produce an actuator activation matrix. The one or more actuator control signals may be, may include, or may correspond with, the actuator activation matrix.

[0362] According to some examples, the set of one or more sensory objects created in the reference environment may be, or may include, one or more light objects. In some such examples, the set of one or more controllable actuators may include one or more light fixtures. In some such examples, the actuator map may be, or may include, a light fixture map. According to some such examples, the actuator activation matrix may be a light activation matrix.

[0363] In some examples, the one or more normalized sensory objects may include one or more normalized light objects that are normalized by the one or more factors. According to some such examples, method 2000 may involve normalizing one or more light fixture positions by the one or more factors.

[0364] In some examples, method 2000 may involve maintaining spatial congruity between a first presentation on a first display screen of the reference environment and a second presentation on a second display screen of the playback environment. According to some such examples, maintaining the spatial congruity may involve stretching or re-sizingD24100WO01coordinates of the second display screen. In some examples, method 2000 may involve warping coordinates from one or more adjacent playback environment surfaces onto a playback environment wall. According to some examples, method 2000 may involve maintaining spatial congruity between a reference environment presentation on one or more screens of the reference environment and a playback environment presentation on two or more screens of the playback environment.

[0365] According to some examples, method 2000 may involve adapting the rendering based on light fixture locations, light fixture density, or both.

[0366] In some examples, the rendering may involve rendering one or more sensory volumetric fields. In some such examples, method 2000 may involve detecting spatial modulations in a sensory volumetric field and normalizing spatial frequencies of one or more normalized sensory objects to retain relative spatial densities across all dimensions of the playback environment.

[0367] According to some examples, the rendering may involve rendering one or more volumetric textures. In some such examples, method 2000 may involve spatial resampling of the one or more volumetric textures and normalizing the spatial resampling.

[0368] In some examples, the rendering may involve rendering the one or more normalized sensory objects relative to a domain bound by one or more actuators.Multi-Modal Multi-Stream Player and Rendering System

[0369] This section of the present disclosure describes a multi-modal multi-stream player and rendering system capable of simultaneously rendering an arbitrary number of sensory object bitstreams, which in some instances may be multi-modal sensory object bitstreams. In some implementations, a multi-modal multi-stream player can be configured with an arbitrary number of modules that will be referred to herein as “stream instances.’’ Each stream instance may, for example, be implemented via instructions (e.g., software) stored on one or more computer-readable and non-transitory media.

[0370] Figure 21 shows example elements of a system for simultaneously rendering a plurality of multi-modal bitstreams. In Figure 21 and the corresponding description, a “multi-modal bitstream” may include a single type of sensory data — such as “bitstream 2” of Figure 21 — or multiple types of sensory data. As with other figures provided herein, the types and numbers of elements shown in Figure 21 are merely provided by way of example. Other implementations may include more, fewer and / or different types and numbers of elements. According to some examples, system 2100 may be, or may include, one or moreD24100WO01devices configured for performing at least some of the methods disclosed herein. In some examples, system 2100 may include one or more instances of the control system 110 of Figure 1 A that are configured for performing at least some of the methods disclosed herein. According to some examples, system 2100 may be an instance of the system 300 that is described with reference to Figure 3.

[0371] According to this example, the system 2100 includes the following elements:• 2100: System for simultaneously rendering a plurality of multi-modal bitstreams; • 2101A: Multi-modal bitstream 1 containing multiplexed audio, video, and MS data;• 2101B: Multi-modal bitstream 2 containing a second type of MS data;• 2101C: Further plurality of multi-modal bitstreams;• 2101D: Multi-modal bitstream N containing other multiple additional types of MS data;• 2102A: Audio elemental stream 1;• 2103 A: Video elemental stream 1;• 2104A: MS elemental stream 1;• 2104B: MS elemental stream 2;• 2104D: MS elemental stream 3;• 2104E: MS elemental stream N;• 2106A: Stream 1 demuxer;• 2106B: Stream 2 demuxer;• 2106D: Stream N demuxer;• 2107A: MS stream 1 decoder:• 2107B: MS stream 2 decoder;• 2107D: MS stream 3 decoder;• 2107E: MS stream N decoder;• 2108A: Audio stream 1 decoder;• 2109A: Video stream 1 decoder;• 2110: Multi-stream multi-sensory Tenderer, which may be an instance of the multi- sensory Tenderer 001 of Figure 3. In this example, the multi-sensory Tenderer 2110 uses actuator information 2111 and decoded sensory bitstreams to drive a plurality of actuators 2114 either directly or by MS controllers 2113;• 2111: Actuator information, which may be an instance of the environment and actuator data 004 of figure 3 - contains information on where actuators are physicallyD24100WO01located within the environment, actuator type information, etc. For light fixtures, the actuator information may include visibility information describing how each light is visible to the viewer(s);• 2112: Actuator control signals• 2113: MS controllers• 2114: Plurality of actuators under control of MS renderer 2110;• 2115: Multi-stream audio Tenderer, which may be an instance of the audio Tenderer 006 of figure 3• 2116: Plurality of loudspeakers under control of audio Tenderer 2115• 2117: Multi -stream video Tenderer, which may be an instance of the video Tenderer 007 of figure 3• 2118: Plurality of display devices under control of video Tenderer 2117• 2119: Multi-modal multi-stream player, which may be an instance of the experience player 002 of Figure 3;• 2120 A: Multi-modal player stream instance 1;• 2120B: Multi-modal player stream instance 2;• 2120C: Plurality of additional multi-modal player stream instances;• 2120D: Multi-modal player stream instance N;

[0372] According to some examples, each of the multi-modal player stream instances 2120A-2120D is configured to process a separate one of the bitstreams 2101A-2101D, each of which includes one or more types of sensory data. In some examples, one or more of the bitstreams 2101A-2101D may be pre-existing and may be played, for example, as serialized files from a storage medium (e.g. pre-authored linear content). According to some examples, one or more of the bitstreams 2101 A-2101D may be received via a network interface according to a variety of protocols (such as Hypertext Transfer Protocol (HTTP), Transmission Control Protocol (TCP), User Datagram Protocol (UDP), etc.). In some examples, one or more of the bitstreams 2101A-2101D may be dynamically generated.

[0373] In some examples, one or more of the multi-modal player stream instances 2120A-2120D may be configured to play back bitstreams at arbitrary frame rates. In some such examples, one or more of the multi-modal player stream instances 2120A-2120D may be configured to resample a received bitstream if the bitstream has a different frame rate than the player stream instance on which it is received.D24100WO01

[0374] According to some examples, the default behavior of the multi-stream multi-sensory Tenderer 2110 is to render each stream and each sensory modality individually and sequentially. However, some examples of the multi-stream multi-sensory Tenderer 2110 may be configured to implement more advanced rendering schemes. In some instances, bitstreams may have different rendering goals and / or strategics. For example, within the lighting modality, one bitstream may be defined — e.g., according to metadata, according to a defined field in the bitstream, etc. — as a high-priority overlay which should always be rendered as visibly as possible. Light objects from multiple bitstreams may be combined or composited in various manners, such as additive, subtractive, multiplicative, masked, overlaid, or according to other methods that may, for example, be defined in the metadata of the streams themselves.

[0375] Multi-stream rendering is an inherently dynamic process. In some examples, the multi-stream multi-sensory Tenderer 2110 may be configured to change the rendering of one or more bitstreams depending on all the bitstreams considered together. According to some examples, the multi-stream multi-sensory Tenderer 2110 may be configured for crossmodality rendering. For example, the total brightness of all light objects rendered by the multi-stream multi-sensory Tenderer 2110’s may impact the video Tenderer 2117, or vice versa.Interfaces and Protocols

[0376] Both sensory content streams and interactivity APIs can be supported via a variety of interfaces, including networking such as TCP, UDP, HTTP, etc., or other higher-level protocols such as Open Sound Control (OSC), Musical Instrument Digital Interface (MIDI), JavaScript Object Notation-Remote Procedure Call (JSON-RPC), Digital Multiplex (DMX), etc.Bitstream-Level Interactivity

[0377] To enable interactivity for various creative authoring and dynamic system requirements, in some examples the multi-stream multi-sensory Tenderer 2110 may be configured to modify one or more types of sensory object parameters interactively.According to some such examples, the multi-stream multi-sensory Tenderer 2110 may be configured to modify one or more types of sensory object parameters at the bitstream level (e.g., applying to all sensory objects in a bitstream) or sub-stream level (e.g., apply to allD24100WO01sensory objects of a single modality in a single bitstream such as MS 3 data 2104D). These stream parameter modifiers may be included in the MS bitstream or may be provided to the multi-stream multi-sensory Tenderer 2110 during rendering by an interactive API as described in 0382.

[0378] In some such examples, stream parameter modifiers may take priority over individual sensory object parameters. For example, light objects may support stream-level parameters such as intensity, feather size, object size, object position, and / or object color, as well as corresponding object-level parameters. According to some examples, each of these stream-level parameters may be specified as absolute (overriding object-level parameters) or relative (either additively offsetting or multiplicatively scaling object-level parameters or values).

[0379] According to some examples, at least some types of stream parameter modifiers may be specified by a content creator to take effect following a transition function or an interpolation function, e.g., over a time interval after the transition function or the interpolation function. The transition function or the interpolation function may, for example, correspond to a change in video and / or audio content that is being presented. For example, if the content includes gaming content, the transition function or the interpolation function may correspond to a change in the game level, an event in the game (such as a player moving from one virtual room to another), etc. If the content includes movie or television program content, the transition function or the interpolation function may correspond to a scene change. In some examples, the specification of the transition function or the interpolation function may also be specified as part of the stream parameter modifier data, either in the bit-stream or via an interactivity API.State Machine Inside the Player

[0380] Figures 22A and 22B show example states of a state machine that may be implemented by the system of Figure 21 according to some implementations. In these examples, Figure 22A shows state 1 of the state machine 2200 at a first time and Figure 22B shows state 2 of the state machine 2200 at second time. According to these examples, the state machine 2200 is implemented by an instance of the control system 110 of Figure 1 A In some examples, the state machine 2200 may be implemented by an instance of the control system 110 that is also configured to implement the multi-modal multi-stream player 2114 of Figure 21. In these examples, the state machine 2200 includes a state module 2210, aD24100WO01dynamic interactive API module 2215, state machine logic 2220 and assets 2225.According to these examples, the state machine 2200 is shown receiving external inputs 2205. As with other figures provided herein, the types and numbers of elements shown in Figures 22A and 22B are merely provided by way of example. Other implementations may include more, fewer and / or different types and numbers of elements. For example, although no transitions to state 1, or from state 2, are shown, some examples may involve transitions to state 1, from state 2, additional states, etc.

[0381] In some examples, the state module 2210 is configured to describe how a particular experience, corresponding to a particular state (such as State 1) is set up and the assets 2225 corresponding to that state. In some such examples, the state module 2210 may include information regarding the mappings from external inputs 2205 to the assets 2225 via the dynamic interactive API module 2215. The static machine logic module 2220 may, in some examples, describe how to transition from one state to another (such as from State 1 to State 2) to provide a different experience, for example by fading out one or more of the assets 2225, fading in one or more of the assets 2225, changing the set of available assets, changing the mapping of inputs to the assets, changing the available inputs, changing the set of states that can be transitioned to etc. Each state can transition to a plurality of other states. The state machine can contain many potential states. Typical numbers range from a 2 up to hundreds.

[0382] According to some examples, external inputs 2205 to the dynamic interactive API module 2215 may include inputs corresponding to modulating the assets 2225 of a state. Such modulation-type external inputs 2205 may, in some examples, be direct user controls such as a fader. In other examples, the external inputs 2205 may be or may include, one or more sensor inputs, such as inputs from a sensor that indicates engine revolutions per minute (RPM).

[0383] In this example, the state machine logic 2220 is configured to respond to commands or instructions from the dynamic interactive API module 2215, for example commands regarding transition events. In some examples, the state machine logic 2220 includes instructions, such as software, for implementing responses to commands from the dynamic interactive API module 2215. The state machine logic 2220 may be configured to update its state, and may also start, stop or modulate the assets 2225 being provided, if so required. A change in state can result in the dynamic interactive API module 2215 changing, which is why it is referred to as a “dynamic” interactive API module 2215. According to some examples, the state machine logic 2220 may provide instructions to the dynamic interactiveD24100WO01API module 2215, for example if a change in the dynamic interactive API module 2215 would be required for a different state. However, in some examples, the API may be the same for multiple states.

[0384] In this example, the state machine logic 2220 is configured to generate data corresponding to transition events, which arc referred to herein as “transition events 2230.” The transition events 2230 may be generated by the state machine logic 2220 after the state machine logic 2220 determines whether certain logical conditions are met or “true,” e.g., based at least in part on inputs from the dynamic interactive API module 2215. The state machine logic 2220 may, for example, determine whether inputs from the dynamic interactive API module 2215 and the current state mean that the state machine 2200 should enter into another state. In some examples, the state machine logic 2220 may be configured to test multiple logical conditions at any given moment, e.g., should the state machine 2200 transition from state N into state M, O, P or K? In other words, the state machine logic 2220 is not necessarily a linear state machine in which one state can only transition to another. The logical condition(s) tested by the state machine logic 2220 to cause a transition event 2230 may, in some examples, include a plurality of trigger events and other conditions such as user conditions (e.g., a user setting indicating “never enter a scary scene”) and state conditions (the state machine 2200 cannot move into state K from state N). A “trigger event" is when some input to the state machine 2200 is triggered. As noted above, a trigger event may also be a condition that is tested by the state machine logic 2220. There may be many types of trigger events. In some examples, a trigger event can be automatically generated by an event (e.g., asset N just finished playing), or by an elapsed time interval (e.g., it has been 10 minutes since the user made an input). A trigger event may, in some examples, be directly caused by user input (e.g., the user just selected option 3 on the menu).

[0385] The assets 2225 may be one or more effects, such as lighting effects, haptic effects, etc., that correspond to a particular state. According to the example shown in Figure 22A, while in state 1 the assets 2225 of the state machine 2200 include assets A, B and C. After a transition event 2230, the state machine 2200 enters state 2, during which the assets 2225 of the state machine 2200 include assets A, D and E. The transition event 2230 may, for example, correspond to a change in a game level, a scene change in a movie or a television program, etc. Assets B and C may, for example, be lighting and haptic effects, respectively, that correspond with state 1, whereas assets D and E may, for example, be lighting effects that correspond with state 2.D24100WO01

[0386] As noted above, according to some examples each bitstream, whether pre-authored or live-streamed, may be modified interactively, e.g., by using one or more APIs that are implemented by the dynamic interactive API module 2215. In some examples, each of the multi-modal player stream instances 2120A-2120D may include an instance of the state machine 2200 that is configured to maintain the state of any sensory objects that the multimodal player stream instance has received. According to some examples, the state machine 2200 instance — or another module that is implemented by the multi-modal player stream instance — may be configured to interpolate one or more sensory object parameters from a current state to a new state over a specified time interval, for example by using a parameterized transfer function. In some examples, the state machine 2200 instance — or another module that is implemented by the multi-modal player stream instance — may be configured to modify per-stream sensory object parameters in an absolute, relative (offset) or scaled manner. According to some examples, a multi-modal player stream instance may be configured to loop over all or part of a bitstream. For example, a bitstream may contain an instruction which signals the Tenderer to rewind the bitstream to a specific time (e.g., the beginning or some other absolute or relative time) and restart parsing from there. In some such examples, the looping process may be controlled, in part, according to a counter (e.g., evaluate a given subsection of a bitstream N times before continuing).

[0387] The ability to have an arbitrary number of configurable streams as described above unlocks new and novel creative authoring possibilities.

[0388] Figure 23 is a flow diagram that outlines one example of a method that may be performed by an apparatus or system such as those disclosed herein. The blocks of method 2300, like other methods described herein, are not necessarily performed in the order indicated. In some implementation, one or more of the blocks of method 2300 may be performed concurrently. Moreover, some implementations of method 2300 may include more or fewer blocks than shown and / or described. The blocks of method 2300 may be performed by one or more devices, which may be (or may include) one or more instances of control system such as the control system 110 that is shown in Figure 1 A and described above. For example, at least some aspects of method 2300 may be performed by an instance of the control system 110 that is configured to implement the multi-modal multi-stream player of Figure 21.

[0389] In this example, method 2300 involves simultaneously rendering a plurality of sensory data bitstreams. Here, block 2305 involves receiving, by a control system, a first content bitstream including first encoded object-based sensory data. In this example, theD24100WO01first encoded object-based sensory data corresponds to one or more types of first sensory effects to be provided by a first set of one or more sensory actuators in a playback environment. In some examples, the set of one or more controllable actuators may include one or more light fixtures, one or more haptic devices, one or more air flow control devices, or combinations thereof.

[0390] According to this example, block 2310 involves extracting, by the control system, first object-based sensory data from the first content bitstream.

[0391] In this example, block 2315 involves providing, by the control system, the first object-based sensory data to a sensory data Tenderer that is configured to provide one or more actuator control signals to the first set of one or more sensory actuators.

[0392] According to this example, block 2320 involves receiving, by the control system, a second content bitstream including second encoded object-based sensory data. In this example, the second encoded object-based sensory data corresponds to one or more types of second sensory effects to be provided by a second set of one or more sensory actuators in a playback environment. Here, the second set of one or more sensory actuators includes one or more sensory actuators of the first set of one or more sensory actuators. In some examples, receiving the first content bitstream and the second content bitstream may involve receiving one or more streamed content bitstreams, obtaining content from a memory system, obtaining newly-generated content, or combinations thereof.

[0393] In this example, block 2325 involves extracting, by the control system, second object-based sensory data from the second content bitstream. According to this example, block 2330 involves providing, by the control system, the second object-based sensory data the sensory data tenderer. In this example, the sensory data tenderer is further configured to provide one or more actuator control signals to the second set of one or more sensory actuators. According to this example, the first content bitstream is received and processed simultaneously with the second content bitstream.

[0394] In some examples, the first encoded object-based sensory data may include first multiplexed sensory objects and extracting the first object-based sensory data from the first content bitstream may involve demultiplexing the first multiplexed sensory objects. In some such examples, the second encoded object-based sensory data may include second multiplexed sensory objects and extracting the second object-based sensory data from the first content bitstream may involve demultiplexing the second multiplexed sensory objects.

[0395] According to some examples, the first encoded object-based sensory data may include first sensory metadata corresponding to the first multiplexed sensory objects andD24100WO01extracting the first object-based sensory data from the first content bitstream may involve demultiplexing the first multiplexed sensory objects and corresponding first sensory metadata. In some such examples, the second encoded object-based sensory data may include second sensory metadata corresponding to the second multiplexed sensory objects and extracting the second object-based sensory data from the second content bitstream may involve demultiplexing the second multiplexed sensory objects and corresponding second sensory metadata.

[0396] In some examples, method 2300 may involve dynamically modifying, by the sensory data Tenderer, the rendering of one or more first sensory objects of the first object-based sensory data responsive to one or more second sensory objects of the second object-based sensory data, or vice versa.

[0397] According to some examples, method 2300 may involve exposing, by the control system, one or more interactivity Application Programming Interfaces (APIs). In some such examples, one or more of the interactivity APIs may allow an external system to trigger sensory playback. In some examples, one or more of the interactivity APIs may allow an external system to augment sensory playback, to modulate sensory playback, or both. In some examples, one or more of the interactivity APIs may allow an external system to augment or modulate one or more sensory data streams, one or more sensory objects, or combinations thereof. According to some examples, one or more of the interactivity APIs may allow an external system to modify per-stream object parameters in an absolute, offset or scaled manner. In some examples, one or more of the interactivity APIs may allow an external system to modify per-stream object parameters over time intervals, according to interpolation functions, or both.

[0398] In some examples, receiving at least one of the first content bitstream or the second content bitstream may involve receiving multi-stream rendering metadata for dynamically configuring the sensory data Tenderer for multi-stream rendering. The multi-stream rendering metadata may, for example, indicate a content bitstream frame rate, one or more aspects of sensory object property interpolation, one or more rendering priorities, content bitstream hierarchy information, or combinations thereof.

[0399] According to some examples, the first set of one or more sensory actuators may include a first set of one or more light fixtures, the one or more types of first sensory effects include one or more first lighting effects and the first object-based sensory data may include one or more first light objects. In some examples, providing the first object-based sensory data to the sensory data Tenderer may involve providing the one or more first light objects toD24100WO01a lighting Tenderer that is configured to provide one or more actuator control signals to the first set of one or more light fixtures. According to some examples, the second set of one or more sensory actuators may include a second set of one or more light fixtures. The second set of one or more light fixtures may, in some instances, include one or more light fixtures of the first set of one or more light fixtures. In some examples, the one or more types of second sensory effects may include one or more second lighting effects and the second object-based sensory data may include one or more second light objects. In some examples, method 2300 may involve providing the second object-based sensory data to the sensory data Tenderer may involve providing the one or more second light objects to the lighting Tenderer and wherein the lighting Tenderer is configured to provide one or more actuator control signals to the second set of one or more light fixtures.

[0400] In some examples, method 2300 may involve dynamically modifying, by the lighting Tenderer, the rendering of the one or more first light objects responsive to one or more second light objects, or vice versa. According to some examples, at least one of the first content bitstream or the second content bitstream may include encoded video data synchronized with the first object-based sensory data or the second object-based sensory data. In some such examples, method 2300 may involve extracting, by the control system, video data from at least one of the first content bitstream or the second content bitstream and providing, by the control system, the video data to a video Tenderer. According to some such examples, method 2300 may involve dynamically modifying, by the lighting Tenderer, the rendering of the video data responsive to one or more first light objects or one or more second light objects. In some examples, method 2300 may involve dynamically modifying, by the lighting Tenderer, the rendering of one or more first light objects or one or more second light objects responsive to the video data.

[0401] According to some examples, method 2300 may involve resampling the first content bitstream, resampling the second content bitstream, or both. The resampling may involve temporal resampling, spatial resampling, or combinations thereof. In some examples, the resampling may be performed according to a sensory data Tenderer configuration, according to a characteristic of one or more sensory actuators in the playback environment, according to a resolution of a playback environment display, or combinations thereof.

[0402] Various features and aspects will be appreciated from the following enumerated example embodiments (“EEEs"):

[0403] EEE1A. A method for controlling a set of one or more controllable actuators in a playback environment, the method comprising: obtaining, by a control system, actuatorD24100WO01data for the set of one or more controllable actuators and playback environment dimension data indicating a playback environment shape; receiving, by the control system, object-based sensory data including a set of one or more sensory objects created in a reference environment, and reference environment dimension data indicating a reference environment shape; normalizing, by the control system, a size of one or more sensory objects by one or more factors based on the playback environment shape relative to the reference environment shape, to produce one or more normalized sensory objects; rendering, by the control system, the one or more normalized sensory objects to produce one or more actuator control signals; and providing, by the control system, the one or more actuator control signals to one or more controllable actuators of the set of controllable actuators.

[0404] EEE2A. The method of EEE1 A, wherein the actuator data includes an actuator map.

[0405] EEE3 A. The method of EEE2A, wherein rendering the one or more normalized sensory objects to produce one or more actuator control signals involves projecting the set of one or more normalized sensory objects using the actuator map.

[0406] EEE4A. The method of EEE3A, wherein projecting the set of one or more normalized sensory objects using the actuator map produces an actuator activation matrix and wherein the one or more actuator control signals comprise the actuator activation matrix.

[0407] EEE5A. The method of EEE4A, wherein: the set of one or more sensory objects created in the reference environment comprises one or more light objects; the set of one or more controllable actuators includes one or more light fixtures; the actuator map comprises a light fixture map; and the actuator activation matrix comprises a light activation matrix.

[0408] EEE6A. The method of EEE5 A, wherein the one or more normalized sensory objects includes one or more normalized light objects that are normalized by the one or more factors, further comprising normalizing one or more light fixture positions by the one or more factors.

[0409] EEE7A. The method of EEE5 A or EEE6A, further comprising maintaining spatial congruity between a first presentation on a first display screen of the reference environment and a second presentation on a second display screen of the playback environment.

[0410] EEE8A. The method of EEE7A, wherein maintaining the spatial congruity involves stretching or re-sizing coordinates of the second display screen.D24100WO01

[0411] EEE9A. The method of EEE7A or EEE8A, further comprising warping coordinates from one or more adjacent playback environment surfaces onto a playback environment wall.

[0412] EEE10A. The method of any one of EEE7A-9A, further comprising maintaining spatial congruity between a reference environment presentation on one or more screens of the reference environment and a playback environment presentation on two or more screens of the playback environment.

[0413] EEE11 A. The method of any one of EEE5A-10A, further comprising adapting the rendering based on light fixture locations, light fixture density, or both.

[0414] EEE12A. The method of any one of EEE1A-11 A, wherein the rendering further comprises rendering one or more sensory volumetric fields, further comprising: detecting spatial modulations in a sensory volumetric field; and normalizing spatial frequencies of one or more normalized sensory objects to retain relative spatial densities across all dimensions of the playback environment.

[0415] EEE13A. The method of EEE12A, wherein the rendering further comprises rendering one or more volumetric textures, further comprising spatial resampling of the one or more volumetric textures and normalizing the spatial resampling.

[0416] EEE14A. The method of any one of EEE1A-13A, wherein the rendering further comprises rendering the one or more normalized sensory objects relative to a domain bound by one or more actuators.

[0417] EEE15A. The method of any one of EEE1A-14A, wherein the set of one or more controllable actuators includes one or more haptic devices, one or more air flow control devices, or combinations thereof.

[0418] EEE16A. An apparatus configured to implement the method of any one of EEE1A-15A.

[0419] EEE17A. A system configured to implement the method of any one of EEE1 A-15A.

[0420] EEE18A. One or more non-transitory and computer-readable media having instructions stored thereon to control one or more devices to implement the method of any one of EEElA-15A.

[0421] EEE1B. A method of simultaneously rendering a plurality of sensory data bitstreams, the method comprising: receiving, by a control system, a first content bitstream including first encoded object-based sensory data, the first encoded object-based sensory data corresponding to one or more types of first sensory effects to be provided by a first setD24100WO01of one or more sensory actuators in a playback environment; extracting, by the control system, first object-based sensory data from the first content bitstream; providing, by the control system, the first object-based sensory data to a sensory data Tenderer that is configured to provide one or more actuator control signals to the first set of one or more sensory actuators; receiving, by the control system, a second content bitstream including second encoded object-based sensory data, the second encoded object-based sensory data corresponding to one or more types of second sensory effects to be provided by a second set of one or more sensory actuators in a playback environment, the second set of one or more sensory actuators including one or more sensory actuators of the first set of one or more sensory actuators; extracting, by the control system, second object-based sensory data from the second content bitstream; and providing, by the control system, the second object-based sensory data the sensory data Tenderer, wherein the sensory data Tenderer is further configured to provide one or more actuator control signals to the second set of one or more sensory actuators and wherein the first content bitstream is received and processed simultaneously with the second content bitstream.

[0422] EEE2B. The method of EEE1B, wherein: the first encoded object-based sensory data includes first multiplexed sensory objects; extracting the first object-based sensory data from the first content bitstream comprises demultiplexing the first multiplexed sensory objects; the second encoded object-based sensory data includes second multiplexed sensory objects; and extracting the second object-based sensory data from the second content bitstream comprises demultiplexing the second multiplexed sensory objects.

[0423] EEE3B. The method of EEE1B or EEE2B, wherein: the first encoded objectbased sensory data includes first sensory metadata corresponding to the first multiplexed sensory objects; extracting the first object-based sensory data from the first content bitstream comprises demultiplexing the first multiplexed sensory objects and corresponding first sensory metadata; the second encoded object-based sensory data includes second sensory metadata corresponding to the second multiplexed sensory objects; and extracting the second object-based sensory data from the second content bitstream comprises demultiplexing the second multiplexed sensory objects and corresponding second sensory metadata.

[0424] EEE4B. The method of any one of EEE1B-3B, further comprising dynamically modifying, by the sensory data Tenderer, the rendering of one or more first sensory objects of the first object-based sensory data responsive to one or more second sensory objects of the second object-based sensory data, or vice versa.D24100WO01

[0425] EEE5B. The method of any one of EEE1B-4B, further comprising exposing, by the control system, one or more interactivity Application Programming Interfaces (APIs).

[0426] EEE6B. The method of EEE5B, wherein one or more of the interactivity APIs allows an external system to trigger sensory playback.

[0427] EEE7B. The method of EEE5B or EEE6B, wherein one or more of the interactivity APIs allows an external system to augment sensory playback, to modulate sensory playback, or both.

[0428] EEE8B. The method of EEE7B, wherein one or more of the interactivity APIs allows an external system to augment or modulate one or more sensory data streams, one or more sensory objects, or combinations thereof.

[0429] EEE9B. The method of EEE8B, wherein one or more of the interactivity APIs allows an external system to modify per-stream object parameters in an absolute, offset or scaled manner.

[0430] EEE 10B. The method of EEE8B or EEE9B, wherein one or more of the interactivity APIs allows an external system to modify per-stream object parameters over time intervals, according to interpolation functions, or both.

[0431] EEE1 IB. The method of any one of EEE1B-10B, wherein receiving at least one of the first content bitstream or the second content bitstream involves receiving multistream rendering metadata for dynamically configuring the sensory data tenderer for multistream rendering.

[0432] EEE12B. The method of EEE1 IB, wherein the multi-stream rendering metadata indicates a content bitstream frame rate, one or more aspects of sensory object property interpolation, one or more rendering priorities, content bitstream hierarchy information, or combinations thereof.

[0433] EEE13B. The method of any one of EEE1B-12B, wherein: the first set of one or more sensory actuators includes a first set of one or more light fixtures; the one or more types of first sensory effects include one or more first lighting effects; the first object-based sensory data includes one or more first light objects; providing the first object-based sensory data to the sensory data Tenderer comprises providing the one or more first light objects to a lighting Tenderer that is configured to provide one or more actuator control signals to the first set of one or more light fixtures; the second set of one or more sensory actuators includes a second set of one or more light fixtures, the second set of one or more light fixtures including one or more light fixtures of the first set of one or more light fixtures; the one or more types of second sensory effects include one or more second lighting effects; theD24100WO01second object-based sensory data includes one or more second light objects; and providing the second object-based sensory data to the sensory data Tenderer comprises providing the one or more second light objects to the lighting Tenderer and wherein the lighting Tenderer is configured to provide one or more actuator control signals to the second set of one or more light fixtures.

[0434] EEE14B. The method of EEE13B, further comprising dynamically modifying, by the lighting Tenderer, the rendering of the one or more first light objects responsive to one or more second light objects, or vice versa.

[0435] EEE15B. The method of EEE13B or EEE14B, wherein at least one of the first content bitstream or the second content bitstream includes encoded video data synchronized with the first object-based sensory data or the second object-based sensory data, and wherein the method further comprises: extracting, by the control system, video data from at least one of the first content bitstream or the second content bitstream; and providing, by the control system, the video data to a video Tenderer.

[0436] EEE16B. The method of EEE15B, further comprising dynamically modifying, by the lighting Tenderer, the rendering of the video data responsive to one or more first light objects or one or more second light objects.

[0437] EEE17B. The method of EEE15B or EEE16B, further comprising dynamically modifying, by the lighting Tenderer, the rendering of one or more first light objects or one or more second light objects responsive to the video data.

[0438] EEE18B. The method of any one of EEE1B-17B, wherein receiving the first content bitstream and the second content bitstream comprises receiving one or more streamed content bitstreams, obtaining content from a memory system, obtaining newly-generated content, or combinations thereof.

[0439] EEE19B. The method of any one of EEE1B-18B, further comprising resampling the first content bitstream, resampling the second content bitstream, or both, wherein the resampling involves temporal resampling, spatial resampling, or combinations thereof.

[0440] EEE20B. The method of EEE19B, wherein the resampling is performed according to a sensory data Tenderer configuration, according to a characteristic of one or more sensory actuators in the playback environment, according to a resolution of a playback environment display, or combinations thereof.

[0441] EEE21B. An apparatus configured to implement the method of any one of EEE1B-20B.D24100WO01

[0442] EEE22B. A system configured to implement the method of any one of EEE1B-20B.

[0443] EEE23B. One or more non-transitory and computer- readable media having instructions stored thereon to control one or more devices to implement the method of any one of EEElB-20B.

[0444] The above description illustrates various embodiments of the present disclosure along with examples of how aspects of the present disclosure may be implemented. The above examples and embodiments should not be deemed to be the only embodiments, and are presented to illustrate the flexibility and advantages of the present disclosure as defined by the following claims. Based on the above disclosure and the following claims, other arrangements, embodiments, implementations and equivalents will be evident to those skilled in the art and may be employed without departing from the spirit and scope of the disclosure as defined by the claims.

Claims

D24100WO01CLAIMSWhat Is Claimed Is:

1. A method for controlling a set of one or more controllable actuators of a playback environment, the method comprising:obtaining, by a control system, actuator data for the set of one or more controllable actuators;receiving, by the control system, object-based sensory data and sensory object geometry metadata, the object-based sensory data including a set of one or more sensory objects and the sensory object geometry metadata indicating a sensory object shape and a sensory object size of a corresponding sensory object of the set of one or more sensory objects;rendering, by a sensory data Tenderer implemented via the control system, the objectbased sensory data to produce one or more actuator control signals, wherein the rendering is based at least in part on the actuator data and the sensory object geometry metadata; and providing, by the control system, the one or more actuator control signals to one or more controllable actuators of the set of controllable actuators.

2. The method of claim 1, wherein the sensory object shape is a three-dimensional (3D) shape.

3. The method of claim 2, wherein the 3D shape comprises a non-spherical shape.

4. The method of claim 2 or claim 3, wherein the 3D shape comprises a polyhedron shape, an ellipsoidal shape, a cylindrical shape or a bicone shape.

5. The method of any one of claims 2-4, wherein the 3D shape is defined by a set of vertices.

6. The method of any one of claims 1-5, wherein the sensory object geometry metadata includes a rotation parameter.

7. The method of any one of claims 1-6, wherein the sensory object geometry metadata includes feathering information and wherein the feathering information defines a feathering region in which an actuator activation value is between zero and one, wherein one is a maximum value.D24100WO018. The method of claim 7, wherein the feathering information comprises a feathering function to be applied in one dimension, two dimensions or three dimensions.

9. The method of claim 8, wherein the feathering function is based, at least in part, on a normal to one or more sensory object surfaces, on a normal to one or more surfaces that define a feathering volume, or combinations thereof.

10. The method of any one of claims 7-9, wherein the sensory object geometry metadata includes a two-dimensional shape or a three-dimensional shape within which the actuator activation value is one and wherein the two-dimensional shape or the three-dimensional shape resides within the feathering region.

11. The method of any one of claims 1-10, wherein the rendering is based, at least in part, on a pre-computed data structure.

12. The method of claim 11, wherein the data structure comprises an acceleration data structure.

13. The method of any one of claims 1-12, wherein the set of one or more controllable actuators includes one or more light fixtures, one or more haptic devices, one or more air flow control devices, or combinations thereof.

14. The method of any one of claims 1-13, wherein the set of one or more controllable actuators includes a set of one or more light fixtures, wherein the object-based sensory data includes light object data and light object geometry metadata, and wherein the sensory data Tenderer comprises a lightscape renderer configured to produce control signals for the set of one or more light fixtures based at least in part on the light object geometry metadata and the actuator data.

15. The method of claim 14, wherein the actuator data includes extended light activation volume data for one or more extended light fixtures, the extended light activation volume data indicating one or more luminance volumes of the playback environment.

16. The method of claim 15, wherein each of the one or more luminance volumes is a volume of the playback environment in which a corresponding extended light fixture is estimated to cause luminance that is perceivable by a human viewer.D24100WO0117. The method of claim 15 or claim 16, wherein the one or more luminance volumes include one or more volumes illuminated by direct light from the extended light fixture, one or more volumes illuminated by reflected light from the extended light fixture, or combinations thereof.

18. The method of any one of claims 15-17, wherein the extended light activation volume data includes one or more rotation parameters corresponding to rotation of the corresponding extended light fixture.

19. The method of any one of claims 15-18, wherein the rendering involves determining an activation of an extended light fixture by computing an inner product of extended light activation volume data and light object geometry metadata in an activation space.

20. The method of any one of claims 15-18, wherein the light object geometry metadata includes a set of light object coordinates and wherein the rendering involves determining an activation of an extended light fixture by projecting at least one object coordinate of the set of light object coordinates onto an extended light activation volume indicated by the extended light activation volume data.

21. The method of claim 20, wherein the rendering involves determining the activation of the extended light fixture by projecting a set of test light object coordinates of the set of light object coordinates onto the extended light activation volume and evaluating activations of a corresponding set of projected test light object coordinates.

22. An apparatus configured to perform the method of any one of claims 1-21.

23. A system configured to perform the method of any one of claims 1-21.

24. One or more non-transitory and computer-readable media having instructions stored thereon for controlling one or more devices to perform the method of any one of claims 1-

Images