Configuration of a renderer for rendering multi-sensory experiences

Abstract

A method of configuring a sensory data renderer may involve obtaining, by a control system, playback environment data corresponding to a playback environment. The playback environment data may include playback environment geometry data and actuator data for a set of controllable actuators of the playback environment. The actuator data may include actuator position data. The method may involve configuring, by the control system and based at least in part on the playback environment data, the sensory data renderer to render received sensory data and to produce actuator control signals for the set of controllable actuators of the playback environment.

Description

D24103W001CONFIGURATION OF A RENDERER FOR RENDERING MULTI-SENSORY EXPERIENCESCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of priority from U. S. Provisional Application No.63 / 720,662, filed on November 14, 2024, and U. S. Provisional Application No. 63 / 784,905 filed on April 7, 2025, each of which is incorporated by reference herein in its entirety.TECHNICAL FIELD

[0002] The present disclosure relates to providing multi-sensory (MS) experiences, which are also referred to herein as multi-modal (MM) experiences, and is more specifically directed to aspects of configuring an MS renderer.BACKGROUND

[0003] Unless otherwise indicated herein, the approaches described in this section are not prior art to the claims in this application and are not admitted as prior art by inclusion in this section.

[0004] Media content delivery has generally focused on audio and screen-based visual experiences. There has been limited delivery of multi-sensory content due to the bespoke nature of actuation. Luminaires, for example, are used extensively as an expression of art and function for concerts. However, each installation is designed specifically for a unique set of luminaires. Delivering a lighting design beyond the set of fixtures the system was designed for is generally not feasible. Other systems that attempt to deliver light experiences more broadly simply do so by extending the screen visuals algorithmically, but are not specifically authored. Haptics content is designed for a specific haptics apparatus. If another device, such as a game controller, mobile phone or even a different brand of haptics device is used, there has been no way to translate the creative intent of content to the different actuators.SUMMARYD24103W001

[0005] At least some aspects of the present disclosure may be implemented via methods. In some instances, the methods may be implemented, at least in part, by a control system such as those disclosed herein. Some such methods involve configuring a sensory data renderer.

[0006] Some disclosed methods may involve obtaining, by a control system, playback environment data corresponding to a playback environment. In this example, the playback environment data includes playback environment geometry data and actuator data for a set of controllable actuators of the playback environment. According to this example, the actuator data includes actuator position data. In some examples, the set of one or more controllable actuators may include one or more light fixtures, one or more haptic devices, one or more air flow control devices, or combinations thereof.

[0007] Some methods may involve configuring, by the control system and based at least in part on the playback environment data, the sensory data renderer to render received sensory data and to produce actuator control signals for the set of controllable actuators of the playback environment. According to some examples, configuring the sensory data renderer may involve playback environment scale normalization.

[0008] Some methods may involve obtaining, by a control system, reference environment data corresponding to a reference environment. The reference environment data may include reference environment geometry data. In some such examples, configuring the sensory data renderer may be based, at least in part, on the reference environment data.

[0009] In some examples, configuring the sensory data Tenderer may involve configuring bed-actuator mapping functionality. In some such examples, the bed-actuator mapping functionality may involve mapping one or more bed channels of the received sensory data to one or more controllable actuators of the set of controllable actuators of the playback environment. According to some examples, configuring the sensory data Tenderer may involve configuring spatial masking functionality of the sensory data renderer.

[0010] In some examples, the set of controllable actuators may include a set of one or more light fixtures. In some such examples, the sensory data Tenderer may include a lightscape renderer configured to received light-based sensory data and to produce control signals for the set of one or more light fixtures.

[0011] According to some examples, configuring the sensory data renderer may involve configuring the lightscape renderer. In some examples, configuring the lightscape renderer may involve configuring extended lights rendering functionality relating to extended light volumes of one or more playback environment light fixtures of the playback environment. According to some examples, configuring the lightscape renderer may involve configuringD24103W001lux density normalization functionality. In some examples, configuring the lightscape renderer may involve configuring image and video object mapping functionality.

[0012] Some disclosed methods may involve providing, by the control system and via a user interface system, information regarding a current configuration of the sensory data renderer. Some disclosed methods may involve providing, by the control system and via a user interface system, one or more user interfaces for receiving user input for altering an automatic configuration of the sensory data renderer, one or more user interfaces for manually setting one or more output values of the sensory data renderer, or combinations thereof.

[0013] Some or all of the operations, functions and / or methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more computer-readable non-transitory media. Such non-transitory media may include one or more memory devices such as those described herein, including but not limited to one or more random access memory (RAM) devices, read-only memory (ROM) devices, etc. Accordingly, some innovative aspects of the subject matter described in this disclosure can be implemented in one or more computer-readable non-transitory media having software stored thereon.

[0014] At least some aspects of the present disclosure may be implemented via apparatus. For example, one or more devices may be capable of performing, at least in part, the methods disclosed herein. In some implementations, an apparatus may include an interface system and a control system. The control system may include one or more general purpose single- or multi-chip processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic, discrete hardware components, or combinations thereof. The control system may be configured to perform some or all of the disclosed methods.

[0015] Details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages will become apparent from the description, the drawings, and the claims. Note that the relative dimensions of the following figures may not be drawn to scale.BRIEF DESCRIPTION OF THE DRAWINGSD24103W001

[0016] Disclosed embodiments are described, by way of example only, with reference to the accompanying drawings.

[0017] Figure 1 A is a block diagram that shows examples of components of an apparatus capable of implementing various aspects of this disclosure.

[0018] Figure IB shows example elements of an endpoint.

[0019] Figure 2 shows examples of actuator elements.

[0020] Figure 3 shows example elements of a system for the creation and playback of multi-sensory (MS) experiences.

[0021] Figure 4 shows example elements of a multi-sensory (MS) renderer.

[0022] Figure 5 shows example elements of another system for the creation and playback of MS experiences.

[0023] Figure 6 A shows an example light map for a table lamp.

[0024] Figure 6B shows an example of an egocentric light map.

[0025] Figure 7 shows elements of a lightscape renderer according to some examples.

[0026] Figures 8A, 8B, 8C, 8D, 8E and 8F show examples of bed membership activation values for an example playback environment.

[0027] Figures 9A and 9B show examples of screen-anchored sensory objects.

[0028] Figures 10, 11, 12A and 12B show examples of aspect-preserving screen anchoring projections.

[0029] Figure 13 shows a top-down view of a reference room in which content is created with respect to a reference TV and a reference user position.

[0030] Figure 14 depicts a top-down view of a playback environment with multiple screens and users.

[0031] Figures 15 A, 15B and 16 show top-down views of reference rooms.

[0032] Figures 17A, 17B, 170, 17D, 17E, 17F, 17G, 17H, 171, 17J, 17K, 17L, 17M, 17N and 170 show examples of mapping source IDs to physical IDs and logical IDs.

[0033] Figure 18 is a flow diagram that outlines one example of a method that may be performed by an apparatus or system such as those disclosed herein.

[0034] Figures 19A, 19B and 19C show examples of graphical user interfaces (GUIs) that may be provided by a tuning tool, such as a tuning software application.DETAILED DESCRIPTION

[0035] Recently, some of the Applicant’ s inventors have developed methods that involve the creation, delivery and / or rendering of object-based sensory content, which may includeD24103W001spatial sensory objects and corresponding sensory metadata. Object-based sensory content may also be referred to herein as object-based sensory data. This object-based abstraction allows creative intent to be implemented in a format that does not require prior knowledge of the specific types, numbers or locations of actuators of a playback environment, thereby enabling greater flexibility and scalability of actuators across playback environments.

[0036] In some instances it may be desirable to create, deliver and / or render what may be referred to herein as “bed-based sensory content.” As used herein, the term “bed-based” is somewhat similar to what is commonly referred to as “channel-based,” in the sense that each “bed” of bed-based content expressly or implicitly refers to a position, direction or zone in a playback environment. For example, bed-based sensory content may include a left bed, a right bed, a front bed and a back bed, each of which is intended, by the content creator(s), to be played back in a corresponding area or zone of a playback environment. However, bedbased content is different from channel-based content. For example, bed-based content does not require that there is a direct correspondence between channels and actuators, as in there is with channel-based content. In one such example, if channel-based content included content for a left light, and the playback environment had no left light, the channel-based content for the left light would not be played back. However, if bed-based content included content for a left light, and the playback environment had no left light, the bed-based content for the left light could, at least in some instances, be flexibly rendered to one or more nearby light fixtures and played back accordingly. In some examples, it may be desirable to create, deliver and / or render object-based sensory data combined with bed-based sensory content.

[0037] Described herein are techniques related to providing multi-sensory media content. In the following description, for purposes of explanation, numerous examples and specific details are set forth in order to provide a thorough understanding of the present disclosure. It will be evident, however, to one skilled in the art that the present disclosure as defined by the claims may include some or all of the features in these examples alone or in combination with other features described below, and may further include modifications and equivalents of the features and concepts described herein.

[0038] In the following description, various methods, processes and procedures are detailed. Although particular steps may be described in a certain order, such order is mainly for convenience and clarity. A particular step may be repeated more than once, may occur before or after other steps (even if those steps are otherwise described in another order), andD24103W001may occur in parallel with other steps. A second step is required to follow a first step only when the first step must be completed before the second step is begun. Such a situation will be specifically pointed out when not clear from the context.

[0039] In this document, the terms “and”, “or” and “and / or” are used. Such terms are to be read as having an inclusive meaning. For example, “A and B” may mean at least the following: “both A and B”, “at least both A and B”. As another example, “A or B” may mean at least the following: “at least A”, “at least B”, “both A and B”, “at least both A and B”. As another example, “A and / or B” may mean at least the following: “A and B”, “A or B”. When an exclusive-or is intended, such will be specifically noted (e.g., “either A or B”, “at most one of A and B”).

[0040] This document describes various processing functions that are associated with structures such as blocks, elements, components, circuits, etc. In general, these structures may be implemented by one or more processors controlled by one or more computer programs.

[0041] As noted above, media content delivery has generally been focused on audio and video experiences. There has been limited delivery of multi-sensory (MS) content due to the customized nature of actuation.

[0042] This application describes methods for extending the creative palette for content creators, allowing spatial, MS experiences to be created and delivered at scale. Some such methods involve the introduction of new layers of abstraction, in order to allow authored MS experiences to be delivered to different endpoints, with different types of fixtures or actuators. As used herein, the term “endpoint” is synonymous with “playback environment” or simply “environment,” meaning an environment that includes one or more actuators that may be used to provide an MS experience. Such endpoints may include a room, such as the living room of a home, a car, a cinema, a night club or other venue, etc. Some disclosed methods involve the creation, delivery and / or rendering of object-based sensory data, which may include sensory objects and corresponding sensory metadata. This abstraction allows creative intent to be implemented in an object-based format that does not require prior knowledge of the specific controller actuation, thereby enabling greater flexibility and scalability of fixtures and actuators across endpoints. An MS experience provided via object-based sensory data may be referred to herein as a “flexibly-scaled MS experience.”D24103W001

[0043] Figure 1A is a block diagram that shows examples of components of an apparatus capable of implementing various aspects of this disclosure. As with other figures provided herein, the types and numbers of elements shown in Figure 1 A are merely provided by way of example. Other implementations may include more, fewer and / or different types and numbers of elements. According to some examples, the apparatus 101 may be, or may include, a device that is configured for performing at least some of the methods disclosed herein, such as a smart audio device, a laptop computer, a cellular telephone, a tablet device, a smart home hub, etc. In some such implementations the apparatus 101 may be, or may include, a server that is configured for performing at least some of the methods disclosed herein.

[0044] In this example, the apparatus 101 includes at least an interface system 105 and a control system 110. In some implementations, the control system 110 may be configured for performing, at least in part, the methods disclosed herein. In some examples, the control system 110 may be configured for obtaining, via the interface system 105, actuator data for a set of controllable actuators. The set of controllable actuators may, for example, be specific to a particular playback environment. According to some examples, the control system 110 may be configured for obtaining, via the interface system 105, object-based sensory data including a set of sensory objects. In some examples, the object-based sensory data may include object-based sensory metadata corresponding to some or all of the sensory objects. The encoded object-based sensory data may correspond to sensory effects such as lighting, haptics, airflow, one or more positional actuators, or combinations thereof, to be provided by a plurality of sensory actuators in an environment.

[0045] According to some examples, the control system 110 may be configured for implementing a multi-sensory (MS) Tenderer. Accordingly, in some examples, the control system 110 may be configured for rendering the object -based sensory data to produce actuator control signals, wherein the rendering is based at least in part on the actuator data. The MS rcndcrcr also may be referred to herein as a “sensory Tenderer,” because in some instances the MS renderer may be rendering only one type of MS data, such as object-based lighting data. According to some examples, the control system 110 may be configured for sending the actuator control signals to one or more controllable actuators of the set of controllable actuators.D24103W001

[0046] According to some examples, the object-based sensory metadata may include sensory spatial metadata indicating at least a spatial position for rendering the object-based sensory metadata within the environment, an area for rendering the object-based sensory metadata within the environment, or combinations thereof. In some implementations, the object-based sensory metadata does not correspond to any particular sensory actuator in the environment. In some examples, the object-based sensory metadata may include abstracted sensory reproduction information allowing the sensory renderer to reproduce authored sensory effects, which also may be referred to herein as intended sensory effects, via various sensory actuator types, via various numbers of sensory actuators and from various sensory actuator positions in the environment.

[0047] In some examples, the control system 110 may be configured for obtaining, via the interface system 105, local context information. The local context information may include local time of day information, local weather information, local human behavior information, local user input, information regarding one or more viewer preferences, information regarding presence or absence of one or more viewers, ambient light information, viewing environment information, local viewer location information, local device usage information, local viewer activity information, or combinations thereof. In some such examples, the rendering process may be based, at least in part, on the local context information.

[0048] In some examples, the content bitstream also may include encoded audio objects synchronized with the encoded object-based sensory metadata. The audio objects may include audio signals and corresponding audio object metadata. According to some examples, the audio objects may include audio signals and corresponding audio object metadata. The audio object metadata may include at least audio object spatial metadata indicating an audio object spatial position for rendering the audio signals within the environment. In some examples, the MS renderer also may be configured for rendering the audio objects.

[0049] The interface system 105 may include one or more network interfaces and / or one or more external device interfaces (such as one or more universal serial bus (USB) interfaces). According to some implementations, the interface system 105 may include one or more wireless interfaces. The interface system 105 may include one or more devices for implementing a user interface, such as one or more microphones, one or more speakers, a display system, a touch sensor system and / or a gesture sensor system. In some examples,D24103W001the interface system 105 may include one or more interfaces between the control system 110 and a memory system, such as the optional memory system 115 shown in Figure 1A.However, the control system 110 may include a memory system in some instances.

[0050] The control system 110 may, for example, include a general purpose single- or multichip processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, and / or discrete hardware components.

[0051] In some implementations, the control system 110 may reside in more than one device. For example, a portion of the control system 110 may reside in a device within an environment (such as a laptop computer, a tablet computer, a smart audio device, etc.) and another portion of the control system 110 may reside in a device that is outside the environment, such as a server. In other examples, a portion of the control system 110 may reside in a device within an environment and another portion of the control system 110 may reside in one or more other devices of the environment.

[0052] Some or all of the methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include memory devices such as those described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, etc. The one or more non-transitory media may, for example, reside in the optional memory system 115 shown in Figure 1 A and / or in the control system 110. Accordingly, various innovative aspects of the subject matter described in this disclosure can be implemented in one or more non-transitory media having software stored thereon. The software may, for example, include instructions for controlling at least one device to process audio data. The software may, for example, be executable by one or more components of a control system such as the control system 110 of Figure 1 A.

[0053] In some examples, the apparatus 101 may include the optional microphone system 120 shown in Figure 1A. The optional microphone system 120 may include one or more microphones. In some implementations, one or more of the microphones may be part of, or associated with, another device, such as a speaker of the speaker system, a smart audio device, etc.D24103W001

[0054] According to some implementations, the apparatus 101 may include the optional actuator system 125 shown in Figure 1A. The optional actuator system 125 may include one or more loudspeakers, one or more haptic devices, one or more light fixtures, also referred to herein as luminaires, one or more fans or other air-moving devices, one or more display devices, including but not limited to one or more televisions, one or more positional actuators, one or more other types of devices for providing a MS experience, or combinations thereof. The term “light fixture” as used herein refers generally to any actuator that is configured to provide light. The term “light fixture” encompasses various types of light sources, including individual light sources such as light bulbs, groups of light sources such as light strips, light panels such as light-emitting diode (LED) panels, projectors, display devices such as television (TV) screens, etc. A “light fixture” may be moveable, and therefore the word “fixture” in this context does not mean that a light fixture is necessarily in a fixed position in space. The term “positional actuators” as used herein refers generally to devices that are configured to change a position or orientation of a person or object, such as motion simulator seats. Loudspeakers may sometimes be referred to herein as “speakers.” In some implementations, the optional actuator system 125 may include a display system including one or more displays, such as one or more light-emitting diode (LED) displays, one or more organic light-emitting diode (OLED) displays, etc. In some examples wherein the apparatus 101 includes a display system, the optional sensor system 130 may include a touch sensor system and / or a gesture sensor system proximate one or more displays of the display system. According to some such implementations, the control system 110 may be configured for controlling the display system to present a graphical user interface (GUI), such as a GUI related to implementing one of the methods disclosed herein.

[0055] In some implementations, the apparatus 101 may include the optional sensor system 130 shown in Figure 1A. The optional sensor system 130 may include a touch sensor system, a gesture sensor system, one or more cameras, etc.

[0056] This application describes methods for rendering and delivering a flexibly scaled multi-sensory (MS) immersive experience (MS IE) to different playback environments, which also may be referred to herein as endpoints. Such endpoints may include a room, such as the living room of a home, a car, a cinema, a night club or other venue, an AR / VR headset, a PC, a mobile device, etc.D24103W001

[0057] Figure IB shows example elements of an endpoint. In this example, the endpoint is a living room 1001 containing multiple actuators 008, some furniture 1010 and a person 1000 — also referred to herein as a user — who will consume a flexibly-scaled MS experience. Actuators 008 are devices capable of altering the environment 1001 that the user 1000 is in. Actuators 008 may include one or more haptic devices, one or more light fixtures, also referred to herein as luminaires, one or more fans or other air-moving devices, one or more display devices, including but not limited to one or more televisions, one or more positional actuators, one or more other types of devices for providing a MS experience, or combinations thereof.

[0058] The number of actuators 008, the arrangement of actuators 008 and the capabilities of actuators 008 in the space 1001 may vary significantly between different endpoint types. For example, the number, arrangement and capabilities of actuators 008 in a car will generally be different from the number, arrangement and capabilities of actuators 008 in a living room, a night club, etc. In many implementations, the number, arrangement and / or capabilities of actuators 008 may vary significantly between different instances of the same type, e.g., between a small living room with 2 actuators 008 and a large living room with 16 actuators 008. The present disclosure describes various methods for creating and delivering flexibly-scaled MSIEs to these non-homogenous endpoints.

[0059] Figure 2 shows examples of actuator elements. In this example, the actuator is a luminaire 1100, which includes a network module 1101, a control module 1102 and a light emitter 1103. According to this example, the light emitter 1103 includes one or more lightemitting devices, such as light-emitting diodes, which are configured to emit light into an environment in which the luminaire 1100 resides. In this example, the network module 1101 is configured to provide network connectivity to one or more other devices in the space, such as a device that sends commands to control the emission of light by the luminaire 1100. According to this example, the network module 1101 is an instance of the interface system 105 of Figure 1A. In this example, the control module 1102 is configured to receive signals via the network module 1101 and to control the light emitter 1103 accordingly. According to this example, the control module 1102 is an instance of the control system 110 of Figure 1A.

[0060] Other examples of actuators also may include a network module 1101 and a control module 1102, but may include other types of actuating elements. Some such actuators mayD24103W001include one or more haptic devices, one or more fans or other air-moving devices, one or more positional actuators, one or more loudspeakers, one or more display devices, etc.

[0061] Figure 3 shows example elements of a system for the creation and playback of multi-sensory (MS) experiences. As noted elsewhere, the terms “multi-sensory” and “multimodal” arc used synonymously herein. As with other figures provided herein, the types and numbers of elements shown in Figure 3 are merely provided by way of example. Other implementations may include more, fewer and / or different types and numbers of elements. According to some examples, system 300 may be, or may include, one or more devices configured for performing at least some of the methods disclosed herein. In some examples, system 300 may include one or more instances of the control system 110 of Figure 1 A that are configured for performing at least some of the methods disclosed herein.

[0062] According to the examples in the present disclosure, creating and providing an object -based MS Immersive Experience (MSIE) approach involves the application of a suite of technologies for creation, delivery and rendering of object-based sensory data, which may include sensory objects and corresponding sensory metadata, to the actuators 008. Some examples are described in the following paragraphs.

[0063] Object-Based Representation: In various disclosed implementations, multi-sensory (MS) effects are represented using what may be referred to herein as multi-sensory (MS) objects, or simply as “sensory objects.” According to some such implementations, properties such as layer-type and priority may be assigned to and associated with attached to each sensory object, enabling content creators’ intent to be represented in the rendered experiences. Detailed examples of sensory object properties are described below.

[0064] In this example, system 300 includes a content creation tool 100 that is configured for designing multi-sensory (MS) immersive content and for outputting object-based sensory data 005, either separately or in conjunction with corresponding audio data 011 and / or video data 012, depending on the particular implementation. The object -based sensory data 005 may include time stamp information, as well as information indicating the type of sensory object, the sensory object properties, etc. In this example, the object-based sensory data 005 is not “channel-based” data that corresponds to one or more particular sensory actuators in a playback environment, but instead is generalized for a wide range of playback environments with a wide range of actuator types, numbers of actuators, etc. In some examples, the object-based sensory data 005 may include object-based light data, object-based haptic data,D24103W001object-based air flow data, or object-based positional actuator data, object-based olfactory data, object-based smoke data, object based data for one or more other types of sensor effects, or combinations thereof. According to some examples, the object-based sensory data 005 may include sensory objects and corresponding sensory metadata. For example, if the object-based sensory data 005 includes object-based light data, the object-based light data may include light object position metadata, light object color metadata, light object size metadata, light object intensity metadata, light object shape metadata, light object diffusion metadata, light object gradient metadata, light object priority metadata, light object layer metadata, or combinations thereof. In some examples, the object-based sensory data 005 may include time data, such as time stamp information. Although the content creation tool 100 is shown providing a stream of object-based sensory data 005 to the experience player 002 in this example, in alternative examples the content creation tool 100 may produce object-based sensory data 005 that is stored for subsequent use. Examples of graphical user interfaces for a light-object-based content creation tool are described below.Examples of MS Object Properties

[0065] Following is a non-exhaustive list of possible properties of MS objects:• Priority;• Layer;• Mixing Mode;• Persistence;• Effect; and• Spatial Panning Law.

[0066] EffectAs used herein, the “effect” of a MS object is a synonym for the type of MS object. An “effect” is, or indicates, the sensory effect that the MS object is providing. If an MS object is a light object, its effect will involve providing direct or indirect light. If an MS object is a haptic object, its effect will involve providing some type of haptic feedback. If an MS object is an air flow object, its effect will involve providing some type of air flow. As described in more detail below, some examples involve other “effect” categories.

[0067] PersistenceD24103W001Some MS objects may contain a persistence property in their metadata. For example, as an moveable MS object moves around in a scene, the moveable MS object may persist for some period of time at locations that the moveable MS object passes through. That period of time may be indicated by persistence metadata. In some implementations, the MS rcndcrcr is responsible for constructing and maintaining the persistence state.

[0068] LayersAccording to some examples, individual MS objects may be assigned to “layers,” in which MS objects are grouped together according to one or more shared characteristics. For example, layers may group MS objects together according to their intended effect or type, which may include but are not limited to the following:Mood / AmbienceInformational- Punctuational / AttentionAlternatively, or additionally, in some examples, layers may be used to group MS objects together according to shared properties, which may include but are not limited to the following:Color- IntensitySizeShapePosition- Region in spacePriority

[0069] In some examples, MS objects may have a priority property that enables the tenderer to determine which object(s) should take priority in an environment in which MS objects are contending for limited actuators. For example, if multiple light objects overlap with a single light fixture at a time during which all of the light objects are scheduled to be rendered, a tenderer may refer to the priority of each light object in order to determine which light object(s) will be rendered. In some examples, priority may be defined between layers or within layers. According to some examples, priority may be linked to specific properties such as intensity. In some examples, priority may be defined temporally: for example, the most recent MS object to be rendered may take precedence over MS objects that have beenD24103W001rendered earlier. According to some examples, priority may be used to specify MS objects or layers that should be rendered regardless of the limitations of a particular actuator system in a playback environment.

[0070] Spatial Panning LawsSpatial panning laws may define a MS object’s movement across a space, how a MS object affects actuators as it moves between them, etc.

[0071] Mixing ModeThe mixing mode may specify how multiple objects are multiplexed onto a single actuator. In some examples, mixing modes may include one or more of the following:Max mode: select the MS object which activates an actuator the most;Mix mode: mix in some or all the objects according to a rule set, for example by summing activation levels, taking the average of activation levels, mixing color according to activation level or priority level, etc.;MaxNmix: mix in the top N MS objects (by activation level), according to a rule set.

[0072] According to some examples, more general metadata for an entire multi-sensory content file, instead of (or in addition to) per-object metadata may be defined. For example, MS content files may include metadata such as trim passes or mastering environment. Trim Controls

[0073] What are referred to in the context of Dolby Vision™ as “trim controls” may act as guidance on how to modulate the default rendering algorithm for specific environments or conditions at the endpoint. Trim controls may specify ranges and / or default values for various properties, including saturation, tone detail, gamma, etc. For example, there may be automotive trim controls, which provide specific defaults and / or rule sets for rendering in automotive environments, for example guidance that includes only objects of a certain priority or layer. Other examples may provide trim controls for environments with limited, complex or sparse multisensory actuators.

[0074] Mastering EnvironmentD24103W001A single piece of multisensory content may include metadata on the properties of the mastering environment such as room size, reflectivity and ambient bias lighting level. The specific properties may differ depending on the desired endpoint actuators. Mastering environment information can aid in providing reference points for rendering in a playback environment.

[0075] MS Object Renderer: Various disclosed implementations provide a renderer that is configured render MS effects to actuators in a playback environment. According to this example, system 300 includes a MS renderer 001 that is configured to render object-based sensory data 005 to actuator control signals 310, based at least in part on environment and actuator data 004. In this example, the MS renderer 001 is configured to output the actuator control signals 310 to MS controllers 003, which are configured to control the actuators 008. In some examples, the MS renderer 001 may be configured to receive light objects and object -based lighting metadata indicating an intended lighting environment, as well as lighting information regarding a local lighting environment. The lighting information is one general type of environment and actuator data 004, and may include one or more characteristics of one or more controllable light sources in the local lighting environment. In some examples, the MS renderer 001 may be configured to determine a drive level for each of the one or more controllable light sources that approximates the intended lighting environment. According to some examples, the MS renderer 001 (or one of the MS controllers 003) may be configured to output the drive level to at least one of the controllable light sources. Some alternative examples may include a separate renderer for each type of actuator 008, such as one renderer for light fixtures, another renderer for haptic devices, another renderer for air flow devices, etc. In other implementations, a single renderer may be configured as a MS renderer and as an audio renderer and / or as a video renderer. In some implementations, the MS renderer 001 may be configured to adapt to changing conditions. Some examples of MS renderer 001 implementations are described in more detail below.

[0076] The environment and actuator data 004 may include what are referred to herein as “room descriptors” that describe actuator locations (e.g., according to an x,y,z coordinate system or a spherical coordinate system). In some examples, the environment and actuator data 004 may indicate actuator orientation and / or placement properties (e.g., directional and north-facing, omnidirectional, occlusion information, etc.). According to some examples, the environment and actuator data 004 may indicate actuator orientation and / or placementD24103W001properties according to a 3x3 matrix, in which three elements (for example, the elements of the first row) represent spatial position (x,y,z), three other elements (for example, the elements of the second row) represent orientation (roll, pitch, yaw), and three other elements (for example, the elements of the third row) indicate a scale or size (sx, sy, sz). In some examples, the environment and actuator data 004 may include device descriptors that describe the actuator properties relevant to the MS renderer 001, such as intensity range and color gamut of a light fixture, the air flow speed range and direction(s) for an air-moving device, etc.

[0077] In this example, system 300 includes an experience player 002 that is configured to receive object-based sensory data 005’, audio data 011’ and video data 012', and to provide object-based sensory data 005 to the MS renderer 001, to provide audio data 011 to the audio Tenderer 006 and to provide the video data 012 to the video Tenderer 007. In this example, the reference numbers for the object-based sensory data 005’, audio data 011’ and video data 012’ received by the experience player 002 include primes (‘), in order to suggest that the data may in some instances be encoded. Likewise, the object-based sensory data 005, audio data 011 and video data 012 output by the experience player 002 do not include primes, in order to suggest that the data may in some instances have been decoded by the experience player 002. According to some examples, the experience player 002 may be a media player, a game engine or personal computer or mobile device, or a component integrated in an television, DVD player, sound bar, set top box, or a service provider media device such as a Chromecast, Apple TV device, or Amazon Fire TV. In some examples, the experience player 002 may be configured to receive encoded object-based sensory data 005’ along with encoded audio data OIL and / or encoded video data 012’. In some such examples, the encoded object -based sensory data 005’ may be received as part of the same bitstream with the encoded audio data Oi l’ and / or the encoded video data 012’. Some examples are described in more detail below. According to some examples, the experience player 002 may be configured to extract the object-based sensory data 005’ from the content bitstream and to provide decoded object-based sensory data 005 to the MS renderer 001, to provide decoded audio data 011 to the audio Tenderer 006 and to provide decoded video data 012 to the video Tenderer 007. In some examples, time stamp information in the objectbased sensory data 005; may be used — for example, by the experience player 102, the MS renderer 001, the audio Tenderer 106, the video Tenderer 107, or all of them — to synchronizeD24103W001effects relating to the object-based sensory data 005’ with the audio data 111’ and / or the video data 112’, which may also include time stamp information.

[0078] According to this example, system 300 includes MS controllers 003 that are configured to communicate with a variety of actuator types using application program interfaces (APIs) or one or more similar interfaces. Generally speaking, each actuator will require a specific type of control signal to produce the desired output from the Tenderer. According to this example, the MS controllers 003 are configured to map outputs from the MS renderer 001 to control signals for each actuator. For example, a Philips Hue™ light bulb receives control information in a particular format to turn the light on, with a particular saturation, brightness and hue, and a digital representation of the desired drive level. In some alternative examples, the MS renderer 001 also may be configured to implement some or all of the MS controllers 003. For example, the MS renderer 001 also may be configured to implement one or more lighting-based APIs but not haptic-based APIs, or vice versa.

[0079] In some examples, room descriptors also may describe the size and orientation of the playback environment itself, to establish a relative or absolute coordinate system to which all objects are positioned. For example, in a living room a display screen may be regarded as the front, in some instances the front and center, and the floor and ceiling may be regarded as the vertical bounds. In some such examples, the room descriptors also may also indicate bounds corresponding with the left, right, front, and rear, walls relative to the front position. According to some examples, the room descriptor also may be provided in terms of a matrix, such as a 3x3 matrix. This room descriptor information is useful in describing the physical dimensions of the playback environment, for example in physical units of distance such as meters. In some such examples, sensory object locations, sensory object sizes, and sensory object orientations may be described in units that are relative to the room size, for example in a range from - 1 to 1. Room descriptors may also describe a preferred viewing position, in some instances according to a matrix.

[0080] The types, numbers and arrangements of the actuators 008 will generally vary according to the particular implementation. In some examples, actuators 008 may include lights and / or light strips (also referred to herein as “luminaires”), vibrational motors, air flow generators, positional actuators, or combinations thereof.

[0081] Similarly, the types, numbers and arrangements of the loudspeakers 009 and the display devices 010 will generally vary according to the particular implementation. In theD24103W001examples shown in Figure 3, audio data Oil and video data 012 are rendered by the audio Tenderer 006 and the video Tenderer 007 to the loudspeakers 009 and display devices 010, respectively.

[0082] As noted above, according to some implementations the system 300 may include one or more instances of the control system 110 of Figure 1A that arc configured for performing at least some of the methods disclosed herein. In some such examples, one instance of the control system 110 may implement the content creation tool 100 and another instance of the control system 110 may implement the experience player 002. In some examples, one instance of the control system 110 may implement the audio Tenderer 006, the video Tenderer 007, the multi-sensory Tenderer 001, or combinations thereof. According to some examples, an instance of the control system 110 that is configured to implement the experience player 002 may also be configured to implement the audio Tenderer 006, the video Tenderer 007, the multi-sensory Tenderer 001, or combinations thereof.

[0083] Figure 4 shows example elements of a multi-sensory (MS) renderer. As with other figures provided herein, the types and numbers of elements shown in Figure 4 are merely provided by way of example. Other implementations may include more, fewer and / or different types and numbers of elements. According to this example, the MS renderer 001 is an instance of the MS renderer 001 that is described with reference to Figure 3. In some examples, the MS renderer 001 may be implemented by one or more instances of the control system 110 of Figure 1 A.

[0084] According to this example, Figure 4 includes the following elements:• 004: Environment and actuator data, which may be as described with reference to Figure 3;• 005: Object-based sensory data 005, which may be as described with reference to Figure 3;• 423 an actuator map (AM) that indicates the locations of at least the controllable actuators 008 in a particular playback environment;• 450 a projection module configured to project the MS objects of the object-based sensory data 005 based, at least in part, on the AM 423. The MS objects may also be referred to herein as “sensory objects,” because in some instances only one type of sensory object — such as only haptic objects or only light objects — may be present in the object-based sensory data 005. In this example, the projection module 450 isD24103W001configured to project the MS objects based, at least in part, on sensory object metadata, which may include at least sensory object location metadata and sensory object size metadata;• 440 an actuator activation matrix (A AM) that is output by the projection module 450 according to this example. The AAM may, for example, indicate sensory objects, if any, that are currently encompassing a volume within the playback environment corresponding to one or more corresponding actuators. For example, the AAM may indicate whether the light object location metadata and light object size metadata of a light object indicate that a particular light fixture is within a volume of the playback environment corresponding to the location and size of the light object;• 451 a mixing module configured to convert the AAM 440 into actuator control signals, based at least in part on the environment and actuator data and the tenderer configuration data;• 452 optional tenderer configuration data, which may include information regarding one or more settings for the MS tenderer 001 such as settings to indicate the desired dynamism, mode, etc. In some examples, the tenderer configuration data 452 may be changed automatically using context-aware systems such as those described in more detail below; and• 310: actuator control signals, which may be as described with reference to Figure 3.In some examples, the actuator control signals 310 may be sent to individual actuators 008, whereas in other examples the actuator control signals 310 may be sent to MS controllers 003, which may be configured to send appropriate control signals to various types of actuators 008.

[0085] According to some examples, the AAM 440 is a matrix describing “how much” a sensory object projects itself onto each actuator according to the actuator map 423. In some examples, the AAM 440 may be a real matrix of size No by NA, where No represents the number of sensor objects and NA represents the number of controllable actuators in the environment. In this example, the mixing module 451 is configured to produce the actuator control signals 310 based at least in part on the AAM 440 and the environment and actuator data 004. In some examples, the mixing module 451 may be configured to produce the actuator control signals 310 based at least in part on the optional Tenderer configuration data 452. According to some examples, the mixing module 451 may be configured to produce the actuator control signals 310 based at least in part on sensory object metadata — which may beD24103W001received as part of the object-based sensory data 005, as shown in Figure 4 — such as mixing and panning laws associated with at least one sensory object.

[0086] In some examples, the mixing module 451 may be configured to produce the actuator control signals 310 based at least in part on one or more of the following:1. Thresholding elements of the AAM 440;2. Taking the maximum of a particular column of the AAM 440 — in other words, taking the sensory object that activated a particular actuator the most as the output;3. Taking any combination of the top N and performing at least one of the following: o Mixing the objects together in the actuator channel;o Pushing objects to adjacent channels.

[0087] In some embodiments, the projection module 450 may be configured to generate a sensory object image using the sensory object’s spatial coordinates — for example, x,y,z coordinates — and the sensory object’s size to produce In(x,y,z), where Inrepresents the sensory object image for the nthsensor object. Then, in some examples, the projection module 450 may be configured to compute, for every column (actuator index) of the AAM 440, the n‘hrow (object index), by taking the inner product of this object image and the actuator map corresponding to that actuator. The object image, actuator map and dot product may be produced and performed by the projection module 450 in any spatial domain that is convenient including, without limitation, polar, cylindrical or rectangular coordinate systems.

[0088] Some implementations may involve implementing what may be referred to herein as “repellers,” which may be used to avoid potentially undesirable sensory effects, such as lighting effects, which may be caused when sensory objects are positioned in one or more areas of a playback environment. In some such examples, repeller data may be included with the environment and actuator data 004 and the AM 423, and therefore may be part of the information that is input to the projection module 450. In some such examples, the spatial coordinate of the MS objects will be augmented when projecting them onto the AM 423 to produce the AAM 440.Multi-Sensory Rendering Synchronization

[0089] Object-based MS rendering involves different modalities being rendered flexibly to the endpoint / playback environment. Endpoints have differing capabilities according to various factors, including but not limited to the following:D24103W001• The number of actuators,• The modalities of those actuators (e.g., light fixture vs. air flow control device vs.haptic device);• The types of those actuators (e.g., a white smart light vs. a RGB smart light, or a haptic vest vs. a haptic seat cushion) and• The location / layout of those actuators.

[0090] In order to render object-based sensory content to any endpoint, some processing of the object signals, e.g. intensities, colors, patterns etc., will generally need to be done. The processing of each modality’s signal path should not alter the relative phase of certain features within the object signals. For example, suppose that a lightning strike is presented in both the haptics and lightscape modalities. The signal processing chain for the corresponding actuator control signals should not result in a time delay of either type of sensory object signal — haptic or light — sufficient to alter the perceived synchronization of the two modalities. The level of required synchronization may depend on various factors, such as whether the experience is interactive and what other modalities are involved in the experience. Maximum time difference values may, for example, range from approximately 10ms to 100ms, depending on the particular context.HAPTICSRendering of object-based haptics content

[0091] Object-based haptics content conveys sensory aspects of the scene through an abstract sensory representation rather than a channel -based scheme only. For example, instead of defining haptics content as a single-channel time-dependent amplitude signal only, that is in turn played out of a particular haptics actuator such as a vibro-tactile motor in a vest the user wears, object-based haptics content may be defined by the sensations that it is intended to convey. More specifically, in one example, we may have a haptic object representing a collision haptic sensory effect. Associated with this object is:• The haptic object’s spatial location;• The spatial direction / vector of the haptic effect;• The intensity of the haptic effect;• Haptic spatial and temporal frequency data: and• A time-dependent amplitude signal.D24103W001

[0092] According to some examples, a haptic object of this type may be created automatically in an interactive experience such as a video game, e.g. in a car racing game when another car hits a player’ s car from behind. In this example, the MS renderer will determine how to render the spatial modality of this effect to the set of haptic actuators in the endpoint. In some examples, the renderer does this according to information about the following:• The type(s) of haptic devices available, e.g., haptic vest vs. haptic glove vs. haptic seat cushion vs. haptic controller;• The locale of each haptic device with respect to the user(s) (some haptic devices may not be coupled to the user(s), e.g., a floor- or seat-mounted shaker);• The type of actuation each haptic device provides, e.g. kinesthetic vs. vibro-tactile;• The on- and off-set delay of each haptic device (in other words, how fast each haptic device can turn on and off);• The dynamic response of each haptic device (how much the amplitude can vary); • The time-frequency response of each haptic device (what time-frequencies the haptic device can provide);• The spatial distribution of addressable actuators within each haptic device: for example, a haptic vest may have dozens of addressable haptics actuators distributed over the user’s torso; and• The time-response of any haptic sensors used to render closed-loop haptic effects (e.g., an active force-feedback kinesthetic haptic device.

[0093] These attributes of the haptics modality of the endpoint will inform the render how best to render a particular haptic effect. Consider the car crash effect example again. In this example, a player is wearing a haptic vest, a haptic arm band and haptic gloves. According to this example, a haptic shockwave effect is spatially located at the place where the car has collided into the player. The shockwave vector is dictated by the relative velocity of the player’ s car and the car that has hit the player. The spatial and temporal frequency spectra of the shockwave effect arc authored according to the type of material the virtual cars are intended to be made of, amongst other virtual world properties. The Tenderer then renders this shockwave through the set of haptics devices in the endpoint, according to the shockwave vector and the physical location of the haptics devices relative to the user.D24103W001

[0094] The signals sent to each specific actuator are preferably provided so that the sensory effect is congruent across all of the (potentially heterogenous) actuators available. For example, the Tenderer may not render very high frequencies to just one of the haptic actuators (e.g., the haptic arm band) due to capabilities lacking in other actuators.Otherwise, as the shockwave moves through the player’s body, because the haptic vest and haptic gloves the user is wearing do not have the capability to render such high frequencies, there would a degradation of the haptic effect perceived by the user as the wave moves through the vest, into the arm band and finally into the gloves.

[0095] Some types of abstract haptic effects include:• Shockwave effects, such as described above;• Barrier effects, such as haptic effects which are used to represent spatial limitations of a virtual world, for example in a video game. If there are kinesthetic actuators on input devices (e.g., force feedback on a steering wheel or joystick), cither active or resistive, then rendering of such an effect can be done through the resistive force applied to the users input. If no such actuators are available in the endpoint then in some examples vibro-tactile feedback may be rendered that is congruent with the collision of the in-game avatar with a barrier;• Presence, for example to indicate the presence of a large object approaching the scene such as a train. This type of haptic effect may be rendered using a low timefrequency rumbling of some haptic devices’ actuators. This type of haptic effect may also be rendered through contact spatial feedback applied as pressure from aircuffs;• User interface feedback, such as clicks from a virtual button. For example, this type of haptic effect may be rendered to the closest actuator on the body of the user that performed the click, for example haptic gloves that the user is wearing.Alternatively, or additionally, this type of haptic effect may also be rendered to a shaker coupled to the chair in which the user is sitting. This type of haptic effect may, for example, be defined using time-dependent amplitude signals. However, such signals may be altered (modulated, frequency-shifted, etc.) in order to best suit the haptic device(s) that will be providing the haptic effect;• Movement. These haptic effects are designed so that the user perceives some form of motion. These haptic effects may be rendered by an actuator that actually moves the user, e.g. a moving platform / seat. In some examples, an actuator may provide aD24103W001secondary modality (via video, for example) to enhance the motion being rendered; and• Triggered sequences. These haptic effects are characterized mainly by their timedependent amplitude signals. Such signals may be rendered to multiple actuators and may be augmented when doing so. Such augmentations may include splitting a signal in either time or frequency across multiple actuators. Some examples may involve augmenting the signal itself so that the sum of the haptic actuator outputs does not match the original signal.D24103W001Spatial and Non-Spatial Effects

[0096] Spatial effects are those which are constructed in a way that convey some spatial information of the multi-sensory scene being rendered. For example, if the playback environment is a room, a shockwave moving through the room would be rendered differently to each haptic device given its location within the room, according to the position and size of one or more haptic objects being rendered at a particular time.

[0097] Non-spatial effects may, in some examples, target particular locations on the user regardless of the user’s location or orientation. One example is a haptic device that provides a swelling vibration on the users back to indicate immediate danger. Another example is a haptic device that provides a sharp vibration to indicate an injury to a particular body area.

[0098] Some effects may be non-diegetic effects. Such effects are typically associated with user interface feedback, such as a haptic sensation to indicate the user completed a level or has clicked a button on a menu item. Non-diegetic effects may be either spatial or non-spatial.Haptic Device Type

[0099] Receiving information regarding the different types of haptics devices available at the endpoint enables the Tenderer to determine what kinds of perceived effects and rendering strategies are available to it. For example, local haptics device data indicating that the user is wearing both haptic gloves and a vibro-tactile vest — or at least local haptics device data indicating that that haptic gloves and a vibro-tactile vest are present in the playback environment — allows the Tenderer to render a congruent recoil effect across the two devices when a user shoots a gun in a virtual world. The actual actuator control signals sent to the haptic devices may be different than in the situation where only a single device is available. For example, if the user is only wearing a vest, the actuator control signals used to actuate the vest may differ with regard to the timing of the onset, the maximum amplitude, frequency and decay time of the actuator control signals, or combinations thereof.Location of the Devices

[0100] Knowledge of the location of the haptics devices across the endpoint enables the Tenderer to render spatial effects congruently. For example, knowledge of the location of the shaker motors in a lounge enables the Tenderer to produce actuator control signals to each ofD24103W001the shaker motors in the lounge in a way to convey spatial effects such as a shockwave propagating through the room. Additionally, knowledge of where wearable haptics devices, whilst implicit by their type, e.g. a glove is on the user’s hand, may also be used by the Tenderer to convey spatial effects in addition to non-spatial effects.Types of Actuation Provided by Haptic Devices

[0101] Haptic devices can provide a range of different actuations and thus perceived sensations. These are typically classed in two basic categories:1. vibro-tactile, e.g. vibrations; or2. Kinesthetic, e.g., resistive or active force feedback.

[0102] Either category of actuations may be static or dynamic, where dynamic effects are altered in real time according to some sensor input. Examples include a touch screen rendering a texture using a vibro-tactile actuator and a position sensor measuring the user’s finger position(s).

[0103] Moreover, the physical construction of such actuators varies widely and affects many other attributes of the device. An example of this is the onset delay or time-frequency response that varies significantly across the following haptic device types:• Eccentric rotating mass;• Linear resonant actuator;• Piezoelectric actuator; and• Linear magnetic ram.

[0104] The tenderer should be configured to account for the onset delay of a particular haptics device type when rendering signals to be actuated by the haptics devices in the endpoint.The On- and Off-Set Delays of the Haptic Devices

[0105] The onset delay of the haptic device refers to the delay between the time that an actuator control signal is sent to the device and the device’s physical response. The off-set delay refers to the delay between the time that an actuator control signal is sent to zero the output of the device and the time the device stops actuating.D24103W001The Time-Frequency Response

[0106] The time-frequency response refers to the frequency range of the signal amplitude as a function of time that the haptic device can actuate at steady state.The Spatial-Frequency Response

[0107] The spatial-frequency response refers to the frequency range of the signal amplitude as a function of the spacing of actuators of a haptic device. Devices with closely-spaced actuators have higher spatial-frequency responses.Dynamic Range

[0108] Dynamic range refers to the differences between the minimum and maximum amplitude of the physical actuation.Characteristics of Sensors in Closed-Loop Haptics Devices

[0109] Some dynamic effects use sensors to update the actuation signal as a function of some observed state. The sampling frequencies, both temporal and spatial along with the noise characteristics will limit the capability of the control loop updating the actuator providing the dynamic effect.AIRFLOW

[0110] Another modality that some multi-sensory immersive experiences (MSIE) may use is airflow. The airflow may, for example, be rendered congruently with one or more other modalities such as audio, video, light-effects and / or haptics. Rather than only specialized (e.g. channel-based) setups for 4D experiences in cinemas which may include “wind effects,” some airflow effects may be provided at other endpoints that may typically include airflow, such as a car or a living room. Rather than a channel-based system, the airflow sensory effects may be represented as an airflow object that may include properties such as:• Spatial location;• Direction of the intended airflow effect;• Intensity / airflow speed; and / or• Air temperature.D24103W001

[0111] Some examples of air flow objects may be used to represent the movement of a bird flying past. To render to the airflow actuators at the endpoint, the MS renderer 001 may be provided with information regarding:• The type of airflow devices e.g. fan, air conditioning, heating;• The position of each airflow device relative to the user' s location, or relative to an expected of the user;• 1’he capabilities of the airflow device, e.g., the airflow device’s ability to control direction, airflow and temperature;• The level of control of each actuator, e.g., airflow speed, temperature range; and • The response time of each actuator, e.g., how long does it take to reach a chosen speed.Some Examples of Airflow Use in Different Endpoints

[0112] In a vehicle such as a car, the object-based metadata can be used to create experiences such as:• Mimicking “chills down your spine” during a horror movie or gaming piece of content with airflow down the chair;• Simulating the movement of a bird flying past; and / or• Create a gentle breeze in a seascape.

[0113] In the small enclosed space of a typical vehicle, temperature changes may be possible to achieve over relatively shorter periods of time — as compared to temperature changes in a larger environment, such as a living room environment. In one example, the MS renderer 001 may cause an increasing air temperature as a player enters a “lava level” or other hot area during a game. Some examples may include other elements, such as confetti in the air vents to celebrate an event, such as the celebration of a goal made by the user’s favorite football team.

[0114] In a living space or other room, airflow may be synchronized to the breathing rhythm of a guided meditation in one example. In another example, airflow may be synchronized to the intensity of a workout, with increased airflow or decreased temperature as intensity increases. In some examples, there may be relatively less control over spatial aspects during rendering. For example, many existing airflow actuators are optimized for heating and / or air conditioning rather than for providing spatially diverse sensory actuation.D24103W001Combinations of Lights, Airflow and HapticsCar Examples

[0115] In some examples, there may be a user interface on the steering wheel or on a touchscreen near or in the dashboard. According to some examples, the following actuators may be present in the car:1. Individually addressable lights, spatially distributed around the car as follows:o on the dashboard;o under the footwells;o on the doors; ando in the center console.2. Individually controllable air conditioning / heating outlets distributed around the car as follows:o In the front dashboard;o Under the footwells;o In the center console facing the rear seats;o On the side pillars;o In the seats; ando Directed to the windscreens (for defogging).3. Individually controllable seats with vibro-tractile haptics; and4. Individually controllable floor mats with vibro-tactile haptics.

[0116] In this example, the modalities supported by these actuators include the following:• Lights across the individually addressable LEDs in the car, plus the indicator lights on the dash and steering wheel;• Air flow via the controllable air conditioning vents:• Haptics, including:o Steering wheel: tactile vibration feedback;o Dash touchscreen: tactile vibration feedback and texture rendering; and o Seats: tactile vibrations and movement.

[0117] In one example, a live music stream is being rendered to four users sitting in four different seat positions. In this example, the MS renderer 001 attempts to optimize the experience for multiple viewing positions. During the build-up before the artist has taken the stage and the previous acts have finished, the content contains:D24103W001• Interlude music;• Low intensity lighting; and• Haptic content representing the moshing of the crowd.

[0118] In addition to the rendered audio and video stream, the light content contains ambient light objects that arc moving slowly around the scene. These may be rendered using one of the ambient layer methods disclosed herein, for example such that there is no spatial priority given to any user’s perspective. In some examples, the haptic content may be spatially concentrated in the lower time-frequency spectrum and may be rendered only by the vibro-tactile motors in the floor mats.

[0119] According to this example, pyrotechnic events during the music stream correspond to multi-sensory-sensory content including:• Light objects that spatially correspond to the location of the pyrotechnics at the event; and• Haptic objects to reinforce the dynamism of the pyrotechnics via a shockwave effect.

[0120] In this example, the MS renderer 001 renders both the light objects and the haptic objects spatially. Light objects may, for example, be rendered in the car such that each person in the car perceives the light objects to come from the left if the pyrotechnics content is located at the left of the scene. In this example, only lights on the left of the car are actuated. Haptics may be rendered across both the seats and floor mats in a way that conveys directionality to each user individually.

[0121] At the end of the concert the pyrotechnics are present in the audio content and both pyrotechnics and confetti are present in the video content. In addition to rendering light objects and haptic objects corresponding to the pyrotechnics as above, the effect of the confetti firing may be rendered using the airflow modality. Lor example, the individually controllable air flow vents of the HVAC system may be pulsed.Living Room Examples

[0122] In this implementation, in addition to an audio / visual (AV) system that includes multiple loudspeakers and a television, the following actuators and related controls are available in the living room:• A haptics vest that the user — also referred to as a player — is wearing;D24103W001• Haptics shakers mounted to the seat in which the player is sitting;• A (haptics) controllable smart watch;• Smart lights spatially distributed around the room;• A wireless controller; and• An addressable air-flow bar (AFB), which includes an array of individually controllable fans directed to the user (similar to HVAC vents in the front dashboard of a car).

[0123] In this example, the user is playing a first person shooter game and the game contains a scene in which a destructive hurricane moves through the level. As it does so, ingame objects are thrown around and some hit the player. Haptics objects rendered by the MS renderer 001 cause a shockwave effect to be provided through all of the haptics devices that the user can perceive. The actuator control signals sent to each device may be optimized according to the intensity of the impact of the in-game objects, the direction(s) of the impact and the capabilities and location of each actuator (as described earlier).

[0124] At a time before the user is struck by an in-game object, the multi-sensory content contains a haptic object corresponding to a non-spatial rumble, one or more airflow objects corresponding to directional airflow; and one or more light objects corresponding to lightning. The MS tenderer 001 renders the non-spatial ramble to the haptics devices. The actuator control signals sent to each haptics device may be rendered such that the ensemble of actuator control signals across the haptics array is congruent in perceived onset time, intensity and frequency. In some examples, the frequency content of the actuator control signals sent to the smart watch may be low-pass filtered, so that they are congruent with the frequency-limited capability of the vest, which is proximate to the watch. The MS renderer 001 may render the one or more airflow objects to actuator control signals for the AFB such that the air flow in the room is congruent with the location and look direction of the player in the game, as well as the hurricane direction itself. Lightning may be rendered across all modalities as (1) a white flash across lights that arc located in suitable locations, c.g., in or on the ceiling; and (2) an impulsive ramble in the user's wearable haptics and seat shaker.

[0125] When the user is struck by an in-game object, a directional shockwave may be rendered to the haptics devices. In some examples, a corresponding airflow impulse may be rendered. According to some examples, a damage take effect, indicating the amount ofD24103W001damage caused to the player by being struck by the in-game object, may be rendered by the lights.

[0126] In some such examples, signals may be rendered spatially to the haptics devices such that a perceived shockwave moves across the player’s body and the room. The MS renderer 001 may provide such effects according to actuator location information indicating the haptics devices locations relative to one another. The MS renderer 001 may provide the shockwave vector and position according to the actuator location information in addition to actuator capability information. According to some examples, a non-directional air flow impulse may be rendered, e.g., all the air vents of the AFB may be turned up briefly to reinforce the haptic modality. In some examples, at the same time, a red vignette may be rendered to the light strip surrounding the TV, indicating to the player that the player took damage in the game.

[0127] Figure 5 shows example elements of another system for the creation and playback of MS experiences. As with other figures provided herein, the types and numbers of elements shown in Figure 5 are merely provided by way of example. Other implementations may include more, fewer and / or different types and numbers of elements. According to some examples, system 500 may be, or may include, one or more devices configured for performing at least some of the methods disclosed herein. In some examples, system 500 may include one or more instances of the control system 110 of Figure 1 A that are configured for performing at least some of the methods disclosed herein.

[0128] According to this example, the system shown in Figure 5 is an instance of the system shown in Figure 3. In this example, the system shown in Figure 5 is a “lightscape” embodiment in which video, audio and light effects are combined to create the MS experience.

[0129] In this example, system 500 includes a lightscape creation tool 100, which is an instance of the content creation tool 100 that is described with reference to Figure 3. The lightscape creation tool 100 is configured for designing and outputting object-based light data 505’, either separately or in conjunction with corresponding audio data 111’ and / or video data 112’, depending on the particular implementation. The object-based light data 505’ may include time stamp information, as well as information indicating light object properties, etc. In some instances, the time stamp information may be used to synchronizeD24103W001effects relating to the object-based light data 505’ with the audio data 111’ and / or the video data 112’, which also may include time stamp information.

[0130] In this example, the object-based light data 505’ includes light objects and corresponding light metadata. For example, the object-based light data may include light object position metadata, light object color metadata, light object size metadata, light object intensity metadata, light object shape metadata, light object diffusion metadata, light object gradient metadata, light object priority metadata, light object layer metadata, or combinations thereof. Although the content creation tool 100 is shown providing a stream of object-based light data 505’ to the experience player 102 in this example, in alternative examples the content creation tool 100 may produce object-based light data 505’ that is stored for subsequent use. Examples of graphical user interfaces for a light-object-based content creation tool are described below.

[0131] In this example, system 500 includes an experience player 102 that is configured to receive object-based light data 505’, audio data 111’ and video data 112’, and to provide object-based light data 505 to the lightscape Tenderer 501, to provide audio data 111 to the audio Tenderer 106 and to provide video data 112 to the video Tenderer 107. As noted elsewhere herein, the object-based light data 505, the audio data 111 and the video data 112 may include time stamp information that may be used to synchronize MS effects with audio and / or video effects. According to some examples, the experience player 102 may be a media player, a game engine or personal computer or mobile device, or a component integrated in an television, DVD player, sound bar, set top box, or a service provider media device such as a Chromecast, Apple TV device, or Amazon Fire TV. In some examples, the experience player 002 may be configured to receive encoded object-based light data 505’ along with encoded audio data 111’ and / or encoded video data 112’, e.g., as part of the same bitstream with the encoded audio data 111’ and / or the encoded video data 112’. According to some examples, the experience player 102 may be configured to extract the object-based light data 505 from the content bitstream and to provide decoded object-based light data 505 to the lightscape Tenderer 501, to provide decoded audio data 111 to the audio Tenderer 106 and to provide decoded video data 112 to the video Tenderer 107. In some examples, the experience player 002 may be configured to allow control of configurable parameters in the lightscape Tenderer 501, such as immersion intensity. Some examples are described below.D24103W001

[0132] As noted above, according to some implementations the system 500 may include one or more instances of the control system 110 of Figure 1 A configured for performing at least some of the methods disclosed herein. In some such examples, one instance of the control system 110 may implement the lightscape creation tool 100 and another instance of the control system 110 may implement the experience player 002. In some examples, one instance of the control system 110 may implement the audio Tenderer 006, the video Tenderer 007, the lightscape Tenderer 501, or combinations thereof. According to some examples, an instance of the control system 110 that is configured to implement the experience player 002 may also be configured to implement the audio Tenderer 006, the video Tenderer 007, the lightscape Tenderer 501, or combinations thereof.

[0133] In some examples, room descriptors of the environment and light fixture data 104 may describe the size and orientation of the playback environment itself, to establish a relative or absolute coordinate system to which all objects are positioned. Room descriptor information may indicate or describe the physical dimensions of the playback environment, for example in physical units of distance such as meters. In some such examples, sensory object locations, sensory object sizes, and sensory object orientations may be described in units that are relative to the room size, for example in a range from -1 to 1. Room descriptors may also describe a preferred viewing position. For example, in a living room a display screen may be regarded as the front, in some instances the front and center, and the floor and ceiling may be regarded as the vertical bounds. In some such examples, the room descriptors also may also indicate bounds corresponding with the left, right, front, and rear, walls relative to the front position. According to some examples, at least some room descriptor information may be provided as a matrix. In some such examples the matrix may be a 3x3 matrix, with one row or column corresponding to one dimension of a three-dimensional space.

[0134] According to this example, system 500 includes a lightscape Tenderer 501 that is configured to render object-based light data 505 to light fixture control signals 515, based at least in part on environment and actuator data 104. In this example, the lightscape Tenderer 501 is configured to output the light fixture control signals 515 to light controllers 103, which are configured to control the light fixtures 108. The light fixtures 108 may include individual controllable light sources, groups of controllable light sources (such as controllable light strips), or combinations thereof. In some examples, the lightscape Tenderer 501 may be configured to manage various types of light object metadata layers,D24103W001examples of which are provided herein. According to some examples, the lightscape Tenderer 501 may be configured to render actuator signals for light fixtures based, at least in part, on the perspective of a viewer. If the viewer is in a living room, that includes a television (TV) screen, the lightscape Tenderer 501 may, in some examples, be configured to render the actuator signals relative to the TV screen. However, in virtual reality (VR) use cases, the lightscape Tenderer 501 may be configured to render the actuator signals relative to the position and orientation of the user’s head. In some examples, the lightscape Tenderer 501 may receive input from the playback environment — such as light sensor data corresponding to ambient light, camera data corresponding to a person’ s location or orientation, etc. — to augment the render.

[0135] In some examples, the lightscape Tenderer 501 is configured to receive object-based light data 505 that includes light objects and object-based lighting metadata indicating an intended lighting environment, as well as environment and light fixture data 104 corresponding to light fixtures 108 and other features of a local playback environment, which may include, but are not limited to, reflective surfaces, windows, non-controllable light sources, light-occluding features, etc. In this example, the local playback environment includes one or more loudspeakers 109 and one or more display devices 510.

[0136] According to some examples, the lightscape Tenderer 501 is configured to calculate how to excite various controllable light fixtures 108 based at least in part on the object-based light data 505 and the environment and light fixture data 104. The environment and light fixture data 104 may, for example, indicate the geometric locations of the light fixtures 108 in the environment, light fixture type information, etc. In some examples, the lightscape Tenderer 501 may be configured to determine which light fixtures will be actuated based, at least in part, on the position metadata and size metadata associated with each light object, e.g., by determining which light fixtures are within a volume of a playback environment corresponding to the light object’s position and size at a particular time indicated by light object time stamp information. In this example, the lightscapc Tenderer 501 is configured to send light fixture control signals 515 to the light controller 103 based on the environment and light fixture data 104 and the object-based light data 505. The light fixture control signals 515 may be sent via one or more of various transmission mechanisms, application program interfaces (APIs) and protocols. The protocols may, for example, include Hue API, LIFX API, DMX, Wi-Fi, Zigbee, Matter, Thread, Bluetooth Mesh, or other protocols.D24103W001

[0137] In some examples, the lightscape Tenderer 501 may be configured to determine a drive level for each of the one or more controllable light sources that approximates a lighting environment intended by the author(s) of the object-based light data 505. According to some examples, the lightscape Tenderer 501 may be configured to output the drive level to at least one of the controllable light sources.

[0138] According to some examples, the lightscape Tenderer 501 may be configured to collapse one or more parts of the lighting fixture map according to the content metadata, user input (choosing a mode), limitations and / or configuration of the light fixtures, other factors, or combinations thereof. For example, the lightscape Tenderer 501 may be configured to render the same control signals to two or more different lights of a playback environment. In some such examples, two or more lights may be located close to one another. For example, two or more lights may be different lights of the same actuator, e.g., may be different bulbs within the same lamp. Rather than compute a very slightly different control signal for each light bulb, the lightscape Tenderer 501 may be configured to reduce the computational overhead, increase rendering speed, etc., by render the same control signals to two or more different, but closely-spaced, lights.

[0139] In some examples, the lightscape Tenderer 501 may be configured to spatially upmix the object-based light data 505. For example, if the object-based light data 505 was produced for a single plane, such as a horizontal plane, in some instances the lightscape Tenderer 501 may be configured to project light objects of the object -based light data 505 onto an upper hemispherical surface (e.g., above an actual or expected position of the user’s head) in order to enhance the experience.

[0140] According to some examples, the lightscape Tenderer 501 may be configured to apply one or more thresholds, such as one or more spatial thresholds, one or more luminosity thresholds, etc., when rendering actuator control signals to light actuators of a playback environment. Such thresholds may, in some instances, prevent some light objects from causing the activation of some light fixtures.

[0141] In some implementations, the lightscape Tenderer 501 may be configured to adapt to changing conditions. Some examples of lightscape Tenderer 501 implementations are described in more detail below.D24103W001

[0142] Light objects may be used for various puiposes, such as to set the ambience of the room, to give spatial information about characters or objects, to enhance special effects, to create a greater sense of interaction and immersion, to shift viewer attention, to punctuate the content, etc. Some such purposes may be expressed, at least in part, by a content creator according to sensory object metadata types and / or properties that arc generally applicable to various types of sensory objects — such as object metadata indicating a sensory object’s location and size.

[0143] For example, the priority of sensory objects, including but not limited to light objects, may be indicated by sensory object priority metadata. In some such examples, sensory object priority metadata is taken into account when multiple sensor objects map to the same fixture(s) in a playback environment at the same time. Such priority may be indicated by light priority metadata. In some examples, priority may not need to be indicated via metadata. For example, the MS renderer 001 may give priority to sensory objects — including but not limited to light objects — that are moving over sensory objects that are stationary.

[0144] A light object may, depending on its location and size and the locations of light fixtures within a playback environment — potentially cause the excitation of multiple lights. In some examples, when the size of a light object encompasses multiple lights, the Tenderer may apply one or more thresholds — such as one or more spatial thresholds or one or more luminosity thresholds — to gate objects from activating some encompassed lights.Examples of Using a Lighting Map

[0145] In some implementations a lighting map, which is an instance of the of the actuator map (AM) that includes a description of lighting in a playback environment, may be provided to the lightscape Tenderer 501. In some such examples, the environment and light fixture data shown in Figure 5 may include the lighting map. According to some examples, the lighting map may be allocentric, e.g., indicating absolute spatial coordinate-based light fall-off, whereas in other examples the lighting map may be egocentric, e.g., a light projection mapped onto a sphere at an intended viewing position and orientation. In the case of a sphere, the lighting map may, in some examples, be projected onto a two-dimensional (2D) surface, e.g., in order to utilize 2D image textures in processing. In any case, the lighting map should indicate the capabilities and the lighting setup of the playback environment, such as a room. In some embodiments the lighting map may not directly relateD24103W001to physical room characteristics, for example if certain user preference-based adjustments have been made.

[0146] In some examples, there may be one lighting map per light fixture, or per light, in a playback environment. According to some examples, the intensity of light indicated by the light map may be inversely correlated to the distance to the center of the light, or may be approximately (e.g., within plus or minus 5%, within plus or minus 10%, within plus or minus 15%, within plus or minus 20%, etc.) inversely correlated to the distance to the center of the light. The intensity values of the light map may indicate the strength or impact of the light object onto the light fixture. For example, as a light object approaches a lightbulb, the lightscape Tenderer 501 may be configured to determine that the lightbulb intensity will increase as the distance between the light object and the lightbulb decreases. The lightscape Tenderer 501 may be configured to determine the rate of this transition based, at least in part, on the intensity of light indicated by the light map.

[0147] Figure 6A shows an example light map for a table lamp. The area 602 indicates the location of the lamp, mapped onto polar coordinates from the main viewer position. The lightscape Tenderer 501 is, in some examples, configured to map lightscape objects into a common rendering space, which may be allocentric or egocentric. Figure 6B shows an example of an egocentric light map. In this example, Figure 6B shows mapping for a spot light object onto a sphere at an intended viewing position and orientation. In Figures 6A and 6B, darker areas are indicated by dots that are relatively closer together, whereas brighter areas are indicated by dots that are relatively farther apart.

[0148] Inside this common rendering space, in some examples the lightscape Tenderer 501 may be configured to use a dot product multiplication between a light object and the light map for each light to compute a light activation metric, e.g., as follows:LM ■ Obj~ min(E LM, £ Objj

[0149] In the foregoing equation, Y represents the light activation metric, LM represents the lighting map and Obj represents the map of a light object. The light activation metric indicates the relative light intensity for the actuator control signal output by the lightscape Tenderer 501 based on the overlap between the light object and the spread of light from the light fixture. In some examples, the lightscape Tenderer 501 may use the maximum or closest distance, or other geometric metrics, from the light object to the light fixture as partD24103W001of the determination of light intensity. In some implementations, instead of computing the light activation metric, the lightscape Tenderer 501 may refer to a look-up-table to determine the light activation metric.

[0150] The lightscape Tenderer 501 may repeat one of the foregoing procedures for determining the light activation metric for all light objects and all controllable lights of the playback environment. Thresholding for light objects that produce a very low impact on light fixtures may be helpful to reduce complexity. For example, if the effect of a light object would cause an activation of less than a threshold percent of light fixture activation — such as less than 10%, less than 5%, etc. — the lightscape Tenderer 501 may disregard the effect of that light object.

[0151] The lightscape Tenderer 501 may then use the resultant light activation matrix Y, along with various other properties such as the chosen panning law (either indicated by light object metadata or Tenderer configuration) or the priority of the light object, to determine which objects get rendered by which lights and how. Rendering lights-objects into light fixture control signals may involve:• Altering the luminance of a light-object as a function of the distance it is from the light fixture;• Mixing the colors of multiple light-objects that are simultaneously (multiplexed) rendered by a single light fixture: or• Altering either of the above based on the light object priority.Some detailed examples are disclosed herein.RENDERING PARAMETERS

[0152] In addition to the information carried by the light object metadata, the rendering of light-objects can be a function of the settings or parameters of the lightscape Tenderer 501 itself. These may include:• Velocity priority - when this parameter is set, light objects that are moving are given a higher priority than those which are not. Having the velocity priority parameter set enhances the dynamism of the rendered scene;• Color priority - light-objects with higher saturation values will take priority;• Activation threshold - the minimum light activation, Y, that must be achieved in order to activate a light-fixture;D24103W001• Accessibility - certain colors may be chosen over others to best represent the experience for colorblind users. Certain flash rates may be avoided for those with photo-sensitivities.RENDERING CONFIGURATION (MODES)

[0153] In addition to the information earned by the light object metadata, the lightscape Tenderer 501 may, in some implementations, be configured according to different modes. As used herein, the term “mode” is different from “parameter” in the sense that modes may, for example, involve completely different signal paths, whereas parameters may simply parameterize these signal paths. For example, one mode may involve the projection of all light objects onto a lighting map before determining how / what to render to the light-fixtures, while another mode may only snap the highest-priority lights to the nearest light fixtures. Modes may include:• Modes to support low light-fixture count. In these modes, the rendering parameters and the light object metadata are utilized in order to determine which subset of lightobjects are to be rendered and in what manner. Here, the “manner” refers to the tradeoff between the spatial, color, temporal fidelity of the most prominent light-objects in the scene;• Modes to support different content types, such as music vs. gaming;• Modes in which multiple light objects may be rendered by a single light fixture (or a single light) with color mixing;• Modes in which only a single light object can be rendered by a single light fixture (or a single light);• Modes in which the luminance of the light object is altered as a function of the geometric - or otherwise - distance between the light object and light fixture.COLOR MIXING AND PRIORITIZATION

[0154] Some implementations of the lightscape Tenderer 501 may implement one or more color mixing methods, prioritization methods, or combinations thereof. For example, when there are multiple light objects that simultaneously influence the same light fixture(s), the lightscape Tenderer 501 may implement a color prioritization algorithm. In the simplest embodiment, only one light object influences the light fixture. The particular light object that will affect the light fixture may, in some examples, be determined by one of the following criteria: (1) the light object that is closest to the light fixture, which may beD24103W001referred to as “snap-to-color”; (2) the light object that has the highest percentage of its impact on that light fixture, (3) the light object with the highest priority pre-defined, (4) the light object that is the brightest. In each of these conditions, only the reference color (potentially at a diminished brightness) may be shown in some instances.

[0155] In some instances it may be desirable to mix lighting. For example, when mimicking the physical characteristics of having multiple colored light fixtures with a single light fixture, modeling the physics of light mixing can make rendering more realistic. This is best done in a physics based, perceptually uniform, color space such as XYZ. In some examples, light mixing may be a linear process within a color space. According to some examples, when there are multiple light objects that simultaneously influence the same light fixture(s), light mixing may involve mixing color and adding intensity.

[0156] According to some examples, the lightscape Tenderer 501 may be configured to calculate the light mixing of two light objects as follows:XTZ„ew= a * XYZ + p * XYZ2

[0157] In this example, the mixing occurs in the XYZ color space. In the foregoing expression, XYZnewrepresents the result of light mixing, XYZrrepresents the color of a first light object, a represents a constant that indicates the weighting of first light object’ s color, XYZ2represents the color of a second light object and represents a constant that indicates the weighting of second light object’s color. The alpha and beta values may, for example, correspond to the amount of light intensity falloff due to the distance from each light object to the light fixture. The alpha and beta values may, for example, be extracted from the lighting map or from some other, potentially geometric, model.

[0158] In cases where physical modeling may be too computationally expensive, faster methods will allow for approximations. In one embodiment, the lightscape Tenderer 501 may be configured to calculate the light mixing of two light objects in the HSV color space, as follows:HSVnew= [a * HS + p * HS2IV + V2]In the foregoing expression, H represents hue and S represents saturation, HS1represents the color of a first light object, a represents a constant that indicates the weighting of first light object’s color, Vi represents the intensity of the first light object, HS2represents the color ofD24103W001a second light object, P represents a constant that indicates the weighting of second light object’s color and V2 represents the intensity of the second light object. In this example, the hue (H) and the saturation (S) are scaled by alpha and beta based on the amount they contribute at the rendered light fixture location. Then the intensities (V) are added to model the addition of light sources.

[0159] A typical lightscape scene may have anywhere from a few light objects up to a few dozen light objects that are active at any given time. A content creator will generally want to have control over the way that these light objects interact with one another within the lightscape Tenderer 501 as they are rendered onto the light fixtures in the endpoint. This may be summarized as the creator wanting to control (1) the relative priority of light objects and (2) the way in which properties of light objects can and cannot be mixed together. The latter is the larger departure from object-based audio rendering procedures, because the lightscape Tenderer 501 normally cannot simply mix the effects of multiple light objects to compute actuator control signals for one actuator. When there are a large number of light objects (even just 3), in combination with the projection of color from the object spatial domain onto the light-fixtures which introduces some warping, the color resulting from simply mixing the effects of multiple light objects would generally not represent the creative intent or preserve the fidelity of the scene. Furthermore, we are restricted to the limited output capacity of the light-fixtures in the endpoint.

[0160] The foregoing issues highlight the importance of providing the content creator with some control over the actions of the lightscape Tenderer 501. Various disclosed examples provide the content creator with the ability to define the light objects layer, the priority on that layer, to control how light objects are mixed on a layer, how layers are mixed, or combinations thereof.

[0161] Figure 7 shows elements of a lightscape renderer according to some examples. As with other figures provided herein, the types and numbers of elements shown in Figure 7 are merely provided by way of example. Other implementations may include more, fewer and / or different types and numbers of elements. According to this example, the lightscape Tenderer 501 is an instance of the lightscape Tenderer 501 that is described with reference to Figure 5. In some examples, the lightscape Tenderer 501 may be implemented by one or more instances of the control system 110 of Figure 1 A.D24103W001

[0162] According to this example, the lightscape Tenderer 501 includes the following elements:• 705: Light objects, which are instances of the object -based sensory data 005 disclosed herein;• 004: Environment and actuator data;• 723: A lighting map (LM), which is an instance of the of the actuator map (AM) that includes a description of lighting in a playback environment;• 750: A projection module, which is an instance of the projection module 450 of Figure 4 and is configured to projects the light objects using the LM 723;• 740: A light activation matrix (LAM), which is an instance of the actuator activation matrix (AAM) 440 and is the output of the projection module 750;• 751: A mixing module 751, which is an instance of the mixing module 451 of Figure 4 and is configured to convert the LAM 740 into actuator commands 741;• 741: The actuator commands 741, which in this example are sent directly to the light fixtures — which are instances of the actuators 008 — to control them, but which may in other examples be sent to light controller APIs 103, which will send corresponding control signals to the light fixtures 008;• 752: Renderer configuration data, which may include settings such as the desired dynamism and mode. In some disclosed context-aware examples, the renderer configuration data 752 may be changed automatically;• 702: An intra-layer mixing module, which is configured to mix light objects of the same layer according to a mixing law;• 710: Light activation vectors, in this example a light activation vector 710 for every layer, containing the activation values and mixed colors of every light fixture; and • 703: An inter-layer blending module, which is configured to blend the light activation vectors 710 together to obtain the rendered actuator commands 741.

[0163] According to some examples, the actuator activation matrix (AAM) 440 of Figure 4 or the light activation matrix (LAM) 740 may be a real matrix of size Noby NA. which is denoted as A in the following equation:ao,oaNo-l, NA-l.D24103W001In the foregoing equation, aij represents the activation value of the ithlight object on the jthactuator. The actuators are lights in this example. The ithrow of the A matrix contains all of the lights activated by the ithlight object. The jthcolumn of the A matrix represents all of the light objects activating the jthlight fixture. This is a useful intermediate data product, as the lightscapc rcndcrcr 501 has not yet performed any mixing of light objects, which results in information loss. This has various potential benefits, including the possibility of optimizing the scene by analyzing A and then warping the scene (see examples below).

[0164] In some examples, there may be an A matrix for every layer being processed by the lightscape Tenderer 501. Thus, the lightscape Tenderer 501 can define a tensor of size Noby NAby Nt(dimension ordering is arbitrary), where NLrepresents the number of layers in the lightscape Tenderer 501. In this document A may refer to either the tensor or the matrix form. The context will inform the reader which it is.

[0165] According to some examples, the intra-layer mixing module 702 is configured to mix all of the light objects on a given layer. This process collapses the A matrix (for that layer) into a light activation vector 710 v, of size 1 by NA. The A matrix contains the activation values, not the color values. We can express the intra-layer mixing module 702 in its general form as a function which produces the light activation vector v (710), as follows:v = (A)

[0166] However, in this example the intra-layer mixing process also produces a vector c for every layer in the matrix containing the color mixing result for that layer, so we may alter the equation above as follows:v, c = (A c0)

[0167] In the above equation, () now outputs both the activation vector, v, and the intralayer mixed colors c as a function of A and co, which is a vector of length Nocontaining all of the objects’ colors.

[0168] The interlayer blending process takes all Ntv and c vectors packed into v’ and c’ matrices of size NAby Ntand outputs a single vector o of length NAcontaining the colors for each actuator. This can be written generally as:o = g (y', c'D24103W001

[0169] Examples of () are given in the Example Object Mixing Laws section of this disclosure. Examples ofare given in the Example Layer Blending Laws section of this disclosure.Example Light Activation Laws

[0170] Light activation laws can range from simple geometric projections and distances to complex responses that involve precomputing and storing in a look up table (LUT). An example of the latter is the lighting map. In some examples, the LUT provides an activation value that may be based on the objects size, position, layer, velocity and potentially other parameters. This activation value may not represent how much the actual light or light fixture projects light onto the playback environment. In some examples, the activation value may be optimized to provide a sparse A matrix (which simplifies the mixing and potential object prioritization problems) and is based on such a (measured or simulated) projection.

[0171] Simple geometric-based activation laws may be applied in different reference frames, e.g., an allocentric or an egocentric reference frame. Such activation laws may be applied using various coordinate systems, for example rectangular, spherical or cylindrical coordinate systems.

[0172] Following is an example of an allocentric rectangular activation law:ALGORITHM 1For i=0; i<7Vo; i ++ do; / / loop over all objectsFor j=(); j< NA; j ++ do / / loop over all light-fixturesA[i,j] = 0For i=0; i< Noi ++ do; / / loop over all objectsFor j=(); j< NA; j ++ do / / loop over all light-fixturesd= n rs- di2If d < O?‘zeA[i,j] = 1Else if d < (O^ze+ o[eather)A[i,j] = ( Ofze+ O[eather- d) I (p{eather+ s')D24103W001Algorithm 1 may, for example, be implemented by a control system that is configured to provide an instance of the projection module 750 of Figure 7. In Algorithm 1,represents the ithobject;- Lj represents the jthlight- fixture;d represents the Euclidean distance between the light and object;Oizerepresentsthe size of the object in the lightscape metadata;gfeatherrepresen^sthe feather radius of the object in the lightscape metadata; and e represents a small number, e.g., 10A- 10, for regularization.

[0173] An implementation of an egocentric spherical activation law may be substantially like the above Euclidean implementation, but one in which the coordinates are first transformed so that the user position and orientation define the origin (the light and object positions are references from this position and orientation) and the position and size are now angular. Distances and activation functions that can be used in place of d include, but are not limited to, the Euclidean distance, p-norm distance, cosine distance, logistic function, gaussian activation function, and rectified linear activation.

[0174] Additional variations to activations laws may include:• warping the coordinates of the objects to account for multiple user perspectives (widening the sweet spot); and / or• using arbitrary reference frames, e.g., rotated frames to account for a TV / screen that is not placed orthogonally within the playback environment.Example Object Mixing Laws

[0175] A simple implementation of / t() would be to simply sum over all Norows of A to produce v and using these to weight coto accumulate c. More specifically, for the jth light, v and c may be calculated as follows:i=No-lvUl = A[i,j]i=0i=N0-lc[ / ] = ^[Cj] c0[i]i=0D24103W001

[0176] Some implementations may involve performing a normalization across the columns of A before performing the summation. This normalization may be linear, e.g.,:r., S-:o°-^[M] c0[i]Or nonlinear, for example a SoftMax function:...^i =N0~1eA[i,j]

[0177] However, these normalizations result in the sum of the columns becoming unity, which may not be desirable when performing inter-layer mixing. Some examples involve augmenting the SoftMax function to place it back onto the same range, so that the sum of the columns are equal after the non-linear normalization (which is now a scaling, not a norm), e.g., as follows:...

[0178] The above-described scaling and / or normalization may be performed before, in some instances, only the top N object elements are contributing to the jthlight. Normalization and scaling are motivated by the fact that it may not be desirable to saturate the color c[j]. If we consider an example involving the RGB color model on a range of [0, 1], then the mixing laws above can produce results that exceed 1. In such cases, clipping may be performed in order to send valid RGB codewords to the light fixtures. However, clipping introduces chromaticity errors and, in the worst case when all 3 RGB channels are saturated, the output is white. This is not desirable, which is a further motivator for only taking the top N (for example, N=2) when performing the summations above.

[0179] In some examples, the lightscape Tenderer 501 may be configured for screen mixing. According to some examples, screen mixing may be implemented as follows:ALGORITHM 2For j = 0; j < NAj + +; do / / loop over all light fixtures:c[y] = 0v[j] = 0D24103W001For i = 0; i < No; i + +; do; 11 loop over all objectsc[ / ]+= 1 - (1 - A[i ]c0[i]) (1 - cf / Dv[ / ]+= A[i,j]

[0180] According to some examples, the lightscape Tenderer 501 may be configured for screen mixing with or without activation weighting as exemplified with the screen mixing above.Example Layer Blending Laws

[0181] In some implementations, there is no fundamental difference between the mixing that happens in the intra-layer process and the blending that happens in the inter-layer process. Blending is typically used to refer to the process of combining multiple layers in image processing and computer graphics. Thus, to avoid confusion and to help delineate the intra- and inter-layer processes, the terms “mixing” and “blending” are used herein.

[0182] For blending layers, in one example we may use alpha compositing, specifically A over B alpha compositing. Recall that blending layers is the process of producing the output vector o from the matrices v’ and c’:o-In the foregoing equation, v’ and c’ are of size NAby Nt. for example as follows:■v0,0v0, Ni~l ’-VNA- 1,0VNA-1, N[-1.C0,0 ■"C0, Ni~lIn the foregoing v’ equation,y- represents the net activation value of the light objects mixed into the ithlight- fixture on the j* layer. These light objects are mixed to produce the C;L> J: color.

[0183] In some examples in which the lightscape Tenderer 501 uses multiple layers, the order of the layers may imply some semantics or priority. For example, in some instances there may be an ambient layer, a spatial layer and an overlay layer. In some such examples, the lightscape Tenderer 501 may render these layers in order, in other words the ambientD24103W001layer may be rendered first, then the spatial layer, then the overlay layer. According to some such examples, the lightscape Tenderer 501 may implement methods such as alpha compositing to render these layers and may use the net activation values, v, as a proxy for the alpha values to blend these layers.

[0184] Following is an example of using A over B alpha compositing to implement the function g^) when using an RGB color model.ALGORITHM 3Cbiack = [0,0,0] / / RGB value for blackFor i — 0; i < NA; i + +; do / / loop over all light-fixtureso[i] = c[i, 0]= v[i, 0]For I = 1; I < Ng I + +; do / / loop over layersaa= v[i, I ~ 1]ab= v[i, I]If oacc> 1.0break; / / move on to next light fixturea0= aa+ ab(l — aa)If a0<= 0Cbiackelsecrao[i] + ab(l — aa)c[i, l]&acc+ CCb

[0185] In Algorithm 3, the layers are ordered so that I = 0 has the highest priority and I = Nt— 1 has the lowest.D24103W001Feather Distance

[0186] A feather distance is a distance applied to the sensory objects such that spatial smoothing occurs. It is used to create smooth transitions as objects move around and activate I deactivate actuators. For example, if an object has size = 0.1 and a feather distance (sometimes also called feather size in our docs) of 0.2, then an example activation law would produce an activation value A of:f1’I d - Ofs ze| Qf eather ' Ofze< d < Ofze+ ofeatherI10, otherwised = \\ or - ^°s\\2

[0187] In order to flexibly render sensory data, the Tenderer requires configuration data summarizing the layout and capabilities of the sensory actuators within the playback environment. Some such implementations involve an automatic process in which data about the sensory actuators of the playback environment is processed to produce sensory actuator configuration data. This disclosure details methods and systems for automatic configuration of many different functionalities of a sensory data renderer. Following automatic configuration, a Tenderer tuner or Tenderer user may wish to alter or override the automatic Tenderer configuration results. Accordingly, this disclosure includes methods and systems which allow the performance of the sensory Tenderer — at least the sensory Tenderer functionality that is being tuned at the time — to be visualized. Some such examples involve exposing controls that allow the tuner or user to alter the automatic configuration process results and to override some or all of the automatic configuration parameters. Accordingly, this disclosure details methods and systems for manual and semi-automatic configuration of many different functionalities of sensory data Tenderers, including but not limited to lightscape Tenderers.Flexibly Rendering Bed-Based Sensory Content, Object-Based Sensory Content, or Both

[0188] When rendering spatial objects alone, the above-described actuator activation matrix (AAM) 440 of Figure 4 or the light activation matrix (LAM) 740 of Figure 7 may beD24103W001denoted as Ase 1RNO>< NA, where Norepresents the number of spatial objects and NArepresents the number of actuators, as in the following equation:■ «o,oa0, NA-l 'As —aNo-1,0aN0-l, NA--L

[0189] In the foregoing equation, ai,j represents the activation value of the ithlight object on the jthactuator. The actuators arc lights in this example. The ithrow of the A matrix contains all of the lights activated by the ithlight object. The jthcolumn of the A matrix represents all of the light objects activating the jthlight fixture. This is a useful intermediate data product, as the lightscape Tenderer 501 has not yet performed any mixing of light objects, which results in information loss. This has various potential benefits, including the possibility of optimizing the scene by analyzing A and then warping the scene, as described elsewhere in this disclosure.

[0190] In some examples, there may be an Asmatrix for every layer being processed by the lightscape Tenderer 501. Thus, the lightscape Tenderer 501 can define a tensor of size Noby NAby Nt(dimension ordering is arbitrary), where Ntrepresents the number of layers in the lightscape Tenderer 501. In this document Asmay refer to either the tensor or the matrix form. The context will inform the reader which it is.

[0191] According to some examples, the intra-layer mixing module 702 may be configured to mix all of the light objects on a given layer. In some such examples, this mixing process may involve collapsing the Asmatrix for that layer into a light activation vector 710 — also represented in this document as v — of size 1 by NA. According to some examples, the Asmatrix contains the activation values, but not the actuation codewords (e.g., the color values for lights). We can express the intra-layer mixing module 702 in its general form as a function which produces the light activation vector v (710), as follows:v = fi( s)

[0192] In this example A is set to As, such that only the spatial objects comprise the total activation matrix. However, the reader will note that content such as ambient objects and effects can also produce a spatial activation matrix. In some such examples, the intra-layer mixing process may also produce a vector c for every layer in the matrix containing the codeword mixing result for that layer, so we may alter the equation above as follows:D24103W001v, c = (s, c0)

[0193] In the above equation, f Q now outputs both the activation vector, v, and the intralayer mixed codewords c as a function of Asand co, which is a vector of length Nocontaining all of the objects’ codewords (e.g., colors).

[0194] According to some examples, the interlayer blending process may take allv and c vectors packed into v’G lR'V / lX Nland c’G K1'V / ’X W[matrices and may output a single vector o of length NAcontaining the codewords for each actuator. This can be written generally as:o = g^v'. c")

[0195] In order to support the flexible rendering of bed-based content, some examples may involve constructing — e.g., by a control system — an AAM matrix A that includes both bedbased sensory content and spatial object based sensory content, e.g., as follows:A =kJ

[0196] In the foregoing equation, ABG B, VcX^represents a bed-based actuator activation matrix, where NBrepresents the number of beds. Subsequent mixing and blending processes will result in the bed-based content being flexibly rendered in the endpoint. Bed-based content may be placed on any layer within the Tenderer. Unlike As, which is generally computed by the Tenderer at run-time depending on the spatial object, corresponding spatial object metadata, the types and locations of the actuators in the playback environment, etc., the bed activation matrix ABmay, in some examples, be computed before playback time. For example, the bed activation matrix ABmay, in some examples, be computed as part of a Tenderer configuration process, as part of a Tenderer tuning process, as part of a Tenderer initialization process, etc.

[0197] In order to define AB, the spatial properties of each bed should be defined. One way to define beds is with reference to orthogonal directions, e.g., as follows:• Up• Down• Front• Rear• LeftD24103W001• Right

[0198] These directions may or may not be with reference to a user position, depending on the particular implementation. For example, “up” may refer to the upper half of a reference or playback environment and “down” may refer to the lower half of the reference or playback environment. One such example, in which the playback environment is assumed to be a cube, is referred to as the “cubic-bed heron.”

[0199] There is also a need to define some sort of metric(s) with which to evaluate each actuator and use this to construct AB, which may, in some examples:• Contain only ones and zeros;• Contain only one non- zero element in each column, assuming that columns are bed dimensions. In such examples, each actuator may only be a member of one bed; • Contain multiple non-zero elements in a column, assuming that columns are bed dimensions. In such examples, some actuators may belong to multiple beds;• Contain values which are in the range of [0, I], which is useful in particular when actuators can belong to multiple beds. The values between 0 and 1 provide a sort of mixing function between the beds into each actuator. In such instances, it is useful for the columns of ABto sum to 1.

[0200] A rendering configuration may contain multiple bed-actuator activation matrices AB, though only one bed-actuator activation matrix ABwould normally be used at any given time. Having multiple bed- actuator activation matrices ABcan provide multiple sets of bed definitions, one or more of which may be canonical or “default” bed definitions such as the cubic-bed example noted above, and one or more of which may be defined by the content creator or by a user.

[0201] One way to define a metric in which to evaluate the bed-actuator membership that would be well suited for the cubic-bed example would be simply a squared distance along the normal to the surface of each of the faces of the cubic-beds. This type of method is referred to herein as a “face distance method.” For example, if the playback environment is, or is assumed to be, a room, the face for the “Up” bed would be the ceiling of the room and the face for the “Down” bed would be the floor of the room. In one example, the coordinate system of the playback environment, the reference environment, or both, may be definedD24103W001such that the origin is at the front bottom left-hand corner of the room, the y axis points to the right wall, the x axis points to the rear of the room and the z axis points up. Given the foregoing assumptions, the matrix dX E ]NBX NA, containing the scalar distance from each cubic-bed surface to every actuator, may be computed. For example, for the jthactuator:• dXupJ= |1 - L^S|2.u / *down,j _ |7P°S| I2’.• dXrightJ= |1 -• dXfrontJ= | L^s|2; and. Jy — Il - jP°s\2UArearJ |1| •

[0202] In the foregoing equations, Lr-osE IK3represents a vector containing the x,y,z position of the j,hactuator. One way in which to use the dX matrix to obtain ABis for a control system to determine, or learn, a matrix W E RNB*NAthat is used to compute a matrix z, as follows:z = dX © W

[0203] In the foregoing equation, 0 represents the Hadamard product and z represents a matrix resulting from the Hadamard product of dX and VF. The activation matrix may be found by taking the SoftMax across the activation matrix columns (the bed dimension), as follows:

[0204] The learnable W matrix may be used, e.g., by a control system, to warp the cubic-bed distances dX in order to modify the bed membership of each actuator. There is sufficient expressivity in z in order to realise non-rectangular spatial cost functions that allows such warping methods to be applied to a wide range of playback environments that can vary in shape. According to some examples, in order to learn W, the control systemD24103W001must have some cost function to optimize. The cost function may, for example, be selected to promote one or more desirable properties of the final value of AB. These potentially desirable properties may, for example, correspond with one or more of the following nonlimiting cost functions:• A cost function that penalizes ABmatrices that have all zeros in some rows. That is, some beds have 0 actuators mapping to them;• A cost function that penalizes ABmatrices that have all zeros in some columns. That is, some actuators do not map to beds;• A cost function that penalizes ABmatrices where pairs of columns are not similar whilst the spatial location of those actuators is similar;• A cost function that penalizes ABmatrices where pairs of columns are similar whilst the spatial location of those actuators are not similar;• A cost function that penalizes ABmatrices where columns are not sparse. This type of cost function is useful to promote single bed membership for each actuator;• A cost function that penalizes ABmatrices where columns have large variance between the sum of their rows. This type of cost function is useful to promote similar levels of membership across beds. This type of cost function may be further augmented by weighting each element in the sum by a spatial cost function in order to not over-penalize good ABfor endpoints which are significantly asymmetrical in the distribution of actuators.

[0205] Some implementations may involve defining an actuator membership cost function which will penalize ABmatrices that have all zeros in some rows, e.g., as follows:1 - maxi(ABj')j=0

[0206] In the foregoing equation, ABj represents the jthcolumn of AB. This cost function will promote each actuator having a significant membership in at least one bed.

[0207] Some examples may involve defining a bed membership cost function which will penalize ABmatrices that have all zeros in some columns, e.g., as follows:1 - maxj(AB i)i =0D24103W001

[0208] In the foregoing equation, AB irepresents the Ithrow of AB. This cost function will promote each bed having at least one actuator as a member.

[0209] Some implementations may involve defining a bed equality cost function which will penalize ABmatrices where columns have a large variance between the sum of their rows, e.g., as follows:(NALM= var I — - G\NB

[0210] In the foregoing equation, G e IK, VBrepresents a vector containing the sum of the rows of AB.

[0211] Some examples may involve combining these functions in order to produce an overall cost, e.g., as follows:L =SLS+BLB+MLM

[0212] In the foregoing equation, As.BandMrepresent real scalars. In one example, As. ABand AMmay represent Lagrange multipliers used to weight each cost function to compute the overall cost. Some examples may involve optimizing W iteratively using numerical optimisation methods such as gradient descent. Some examples may involve postprocessing ABafter the iterative optimization process has converged. Some such examples involve post-processing ABby setting the maximum value of each column to 1 and the remaining elements to zero. This will result in each actuator belonging to just one bed, which may be desirable in some instances. In some alternative examples, the “postprocessing” described above may occur during the process of computing the loss functions during the iterative optimization process.

[0213] In some cases, it may be desirable to retain the top M elements of each of the columns of ABif they exceeded some threshold. In some such examples, the control system may set each of the retained elements such that they sum to 1, or to another determined value, but retain their original proportions. In one such example, if the values of a given column of the ABmatrix are [0.1, 0.0, 0.3, 0.6, 0.7, 0.7] and the threshold is 0.5, then the post-processed column is [0.0, 0.0, 0.0, 0.3, 0.35, 0.35]: in this example, the sum of the retained elements that exceeded the 0.5 threshold (0.6, 0.7 and 0.7) is 2 and the retained elements after downward adjustment sum to 1.D24103W001

[0214] Figures 8A, 8B, 8C, 8D, 8E and 8F show examples of bed membership activation values for an example playback environment. As with other figures provided herein, the types and numbers of elements shown in Figures 8A-8F are merely provided by way of examples. Other implementations may include more, fewer and / or different types and / or numbers of elements.

[0215] Here, Figures 8A-8F each show the bed membership activation values of the actuators for a single bed. Figure 8A corresponds to the “up” bed, Figure 8B corresponds to the “down” bed, Figure 8C corresponds to the “left” bed, Figure 8D corresponds to the “right” bed, Figure 8E corresponds to the “front” bed and Figure 8F corresponds to the “rear” bed. According to these examples, actuators (light fixtures) are represented as circles, with actuators represented by a circle having a light outline indicate a membership of 1 (light on) whilst actuators represented by a circle having a dark outline indicate a membership of 0 (light off). These results were obtained using the example cost function given above and then post-processing the resultant ABproduced from the learned IF matrix by setting the maximum value in each column to 1 and setting the remaining elements to 0. In these examples, each actuator is only represented by a circle having a light outline in one of the Figures 8A-8F because each actuator only belongs to one of the beds.

[0216] In some examples, the ABmatrix may be automatically obtained at a rcndcrcr configuration time, which may be a Tenderer initialization time. In some instances, the ABmatrix may be obtained after a Tenderer configuration time, such as when a person manually intervenes to tune the configuration. The tuned configuration may be stored for later use by the Tenderer. According to some alternative examples, there may be no opportunity for manual tuning of an automatic Tenderer configuration.

[0217] A person performing Tenderer configuration or tuning, who may be referred to as a “tuner,” requires control over at least some aspects of the Tenderer configuration process. For the process of producing ABthis control may include, but may not be limited to, the following:1. The ability to explicitly set membership of certain actuators to certain beds;2. The ability to re-mn the optimization process when this explicit set membership acts as a constraint;3. The ability to flag certain beds as not present (in order to not skew the cost functions);D24103W0014. The ability to tune the weights associated with each of the cost functions, particularly the bed membership loss weight;5. The ability to define alternate distance metrics between beds and actuators for the optimization to be based on, for example:a. the ability to choose alternate optimisation methods and / orb. the ability to define custom bed arrangements for personalisation.

[0218] According to some implementations, a tuner may be able to control at least some aspects of the Tenderer configuration process via a tuning tool — such as a tuning software application — which may, in some instances, provide one or more graphical user interfaces (GUI) in order to aid the tuning process with visualisations and feedback.

[0219] Some tuning tools may provide a tuner with the ability to define custom bed arrangements for personalisation. For example, some tuning tools may provide a GUI that allows the tuner to manually select elements of a personalised bed activation matrix and the bed membership each actuator has. In some instances, at least some aspects of bed membership, etc., may be provided by an automatic optimization process, e.g., such as described earlier.

[0220] The tuner may, in some instances, select a set of customized beds — also referred to herein as personalized beds — that provide a way for the end user to control how sensory content is rendered. For example, a set of customised beds may allow a user to choose, via some interface to the Tenderer, where and how sensory content will be rendered in the playback environment. The customised beds may, for example, be selected through an interactive API.Using Beds as a Masking Mechanism

[0221] The bed activation matrix ABmay, in some instances, only have a single non- zero element per column. When this occurs, each actuator maps only to a single bed. An additional utility of a bed activation matrix of this form is the ability to mask objects. According to some examples, in order to use a bed activation matrix of this form to mask objects, one may set the AAM A equal to the masked spatial activation matrix AM, as follows:=AMD24103W001

[0222] In this example, the masked spatial activation matrix AMG IRWo>< Na. In some such examples,=^S,i O AB b

[0223] In the foregoing equation, O represents the Hadamard operator, AM lrepresents the ithcolumn of AM, As irepresents the ithcolumn of As, which represents the actuator activation values for the i,hobject and b, AB bis the bthrow of ABand b represents the bed index that the ithspatial object activation values are being masked by. The bed index b is a parameter that may, for example, be included in a sensory content data stream and may be chosen by the content creator(s) when the content is created.

[0224] In some implementations, a sensory object’s metadata may indicate which row (bed) to use of ABas a mask. In some such examples, a tuner may wish to specify an alternate version of ABfor a given set of beds for the purpose of spatial masking. For example, the tuner may be motivated to make this alternative specification because the tuner has optimized ABfor bed-based content where particular actuators which do not necessarily align spatially close with the bed definitions nonetheless provide sufficient coverage as a set of beds. For example, to ensure all beds have sufficient and equal coverage the tuner may have been willing to compromise spatial fidelity. In such cases, the tuner may want to be able to specify alternate ABjust for the purposes of spatial masking. The tuner may, in some examples, be provided with all of the same controls when building the alternate ABmatrix as the tuner was provided when building ABas described in the previous section.Configuring and Rendering Snapped Spatial Objects

[0225] Due to the nature of the spatial activation functions in the Tenderer, it is possible that some spatial sensory objects do not activate any of the actuators in the endpoint. This can be undesirable for some content creators whose intent is for a spatial sensory object to be rendered in the endpoint even if spatial fidelity is decreased. This intent can be conveyed in metadata associated with a spatial sensory object, e.g., by snapping metadata — also referred to herein as snap metadata — indicating that a corresponding spatial sensory object should be rendered even if spatial fidelity is decreased. The snap metadata may, for example, be implemented by a flag or a bit that is set or not set.

[0226] In some examples, the Tenderer may augment the spatial actuator activation matrix (AAM) if a spatial sensory object has its snap metadata flag set, such that the TendererD24103W001causes one or more actuator control signals to be generated for the corresponding spatial sensory object even if no actuator control signal would otherwise have been generated. The computation of the AAM for a spatially activated object may be preceded by the computation of the offset tensor, d G 1KW°X Na x 3which is the tensor composed of the offset vector, dtj, between all objects and actuators in the light object reference frame. The offset vector di is defined above, as follows:di,j =Rid'ij

[0227] As noted above, d G B,vrepresents the offset vector in the playback environment reference frame and may be determined as follows:d'. = [V°s~ Ofos

[0228] In the foregoing expression, Oosrepresents the position of the ithsensory object and lJj0Srepresents the position of the j,hactuator. The distance matrix D G IRW°X Na, which is defined in the arbitrary spatial shape activation section above, may be computed by taking the L-2 norm across the N index of the d tensor. If the spatial object activation function is isotropic then it will take Di as an argument, otherwise it will take.

[0229] Some activation functions may be non-linear. For example, isotropic activation functions which consume dtj may be spherical. Non-isotropic activation functions may be cuboid, bicone or ellipsoid or spherical (when elements of the size vector are not all the same which means the activation function is spherical by default).

[0230] In some instances, if the distance between the sensory object and the actuator is too great, the actuation value provided by some activation functions may be 0. Fundamentally, this is how sensory objects can fail to be rendered in an endpoint. In order to avoid this, some methods of rendering snapped spatial objects may involve one or more of the following:• Scaling Di j or the elements of dij to reduce their magnitudes:• Augmenting the spatial activation function(s) to account for an additional activation margin.

[0231] In some examples, the control system may render snapped spatial objects with the addition of metadata that is computed ahead of time, e.g., during a Tenderer configuration process and / or a Tenderer tuning process. If the Dtj or the elements of dtj are scaled, then the Tenderer can render snapped spatial objects with knowledge about a snapped spatialD24103W001object’s potential activation across all light fixtures. Rendering a snapped spatial object can be done with information regarding the spatial object’s activation function (e.g., the distance at which the spatial object engages with an actuator). Some examples may involve optimizing the rendering of a snapped spatial object according to the effect of the snapped spatial object on multiple (e.g., all) actuators. Some such examples may involve providing a scaling of activation that is coherent across all of the actuators in a playback environment. In one such example, each of the elements ofor dt may be scaled in a way that is equivalent to augmenting the position vector of the snapped spatial object.

[0232] In some instances, the content creator may want a spatial object to always be present in the final actuation vector. This functionality may be implemented by setting (snapping) the spatial object’s position to be equal to that of the nearest actuator, e.g., as follows:^pOS _ jPOSUi ~Lk

[0233] In the foregoing expression, k is the index of the spatial object with minimal scalar distance and may be expressed as follows:k = argminj (D^ j)

[0234] Some examples also may involve applying a scaling to the j‘hcolumn of d j or Dtj according to precomputed metadata. In some examples, this scaling may be computed ahead of time to ensure that preferred or undesired spatial directions, particular actuators, or both, may be enhanced or penalised accordingly. In some instances, the degree of enhancement or penalization may be a function of the type of the spatial object, of properties of a particular actuator, etc. Moreover, in some examples the argmin() can be completely replaced by using a look up table (LUT) or other data structure (DS), such as an an acceleration data structure, e.g. a binary tree, the values of which may in some examples be computed at Tenderer configuration / tuning / initialisation time. That is, we can omit the expensive computation of DLj or dtj by referring to a DS, e.g., as follows:k = DS(Oos)

[0235] Because we are using the DS to return an index, it is not desirable for any interpolation to occur. In some examples, the DS may be computationally cheap to prepare and evaluate. In some such examples, the DS may simply be a LUT or an array expressed as follows:k = DS Xi.^ Zt]D24103W001

[0236] In some examples, the array may be indexed as follows:fjPOSj.ydyQVOSdz

[0237] In the foregoing expressions, [■] is the floor operator and dx, dy and dz are the spatial resolution of the (x,y,z) dimensions of the DS, respectively. Care should be taken to ensure that the DS is indexed such that (%f,z;) are not out of bounds. This can be done either by clipping the indices, so they are within the size of the DS or clipping the spatial object position to the normalised coordinate space of the endpoint. After the index k has been looked up, in some examples the Tenderer may simply set the lfhcolumn of the i,hrow of A to be 1, effectively snapping the spatial object to that index.

[0238] In order to construct this DS at configuration time we can run an algorithm such as the following:ALGORITHM 1Input Nx, Ny, Nz, NA, £?“[]Output DS[]Function BuildSnapDS (Nx, Ny, Nz, NA, Lpos[] )DS [] <— 0sWx x Nyx Nzfor x <- 0 to Nx— 1 dofor y «- 0 to Nz— 1 dofor z «- 0 to Nz— 1 dod <- InfP «- [x,y,z]Tfor j «- 0 to NA— 1 doe- iip - Lrn"if e < d thend «- eD24103W001DS[x, y,z] =jreturn DS []

[0239] Typical values for Nx, Ny, Nzcan range from 50 to 200, which correspond to isotropic DSs containing 1500 and 8,000,000 elements, respectively. Other DSs may include more or fewer elements. At rendering time, the size of the DS may not be critically important. The size of the DS may be selected based on a trade-off between the desired spatial resolution and the corresponding required memory footprint of the DS. In some cases, it may be desirable for the DS to return a vector of the closest M actuators, which will scale the size of the DS but not affect the computational complexity of evaluating it at render time again. Moreover, some playback environments may have a total number of actuators, NA, in the range of a few 100s to a few 1000s. Consequently, if the DS returns 16-bit values, the 1500 and 8,000,000 DS elements referred to above would require 3000 and 16,000,000 bytes of memory respectively. This would be scaled proportionally by M.

[0240] Whilst the algorithm in ALGORITHM 1 is automatic, in some examples a tuner may control the process of constructing the DS in one or more aspects, such as:The ability to visualize the performance of the DS in a GUI;The ability to manually set which actuator is returned by the DS at a particular index; The ability to set the values of Nx, Ny, Nzand visualise the impact it has on the DS. This may be accomplished by the overlay of a 3D grid onto a visualisation of the actuators in the endpoint. This is especially useful for rooms that are not cubic in shape;- The ability to visualise object trajectories through the endpoint space where the actuators activated by the DS are shown. This may be accomplished by using a GUI with a set of standard object trajectories or test vectors;The ability to tag actuators as not suitable for snapping spatial objects to.Constructing a KD-Tree for Snap-to

[0241] In some examples, a control system — such as a control system implementing a Tenderer — may be configured for snapping a sensory object to an actuator by determining which actuator is closest to the snapping object, e.g., by computing the distance between the sensory object and every actuator in the endpoint. This is obviously a computationally expensive option, because it involves applying “brute force” to obtain the solution.D24103W001

[0242] In some implementations, the control system may be configured to speed up the process of snapping a sensory object to an actuator by using acceleration data structures. One such acceleration data structure which is well-suited to the problem of finding the nearest neighbor is the “kd-tree.” The kd-tree is an algorithm that implements a binary tree structure in which comparisons arc made on different dimensions at different layers of the tree. The canonical implementation evaluates the nthdimension as follows:n = mod(level, k)

[0243] In the foregoing equation, “level” represents the level of the node being tested in the tree and k represents the dimensionality of the data. For implementations involving the disclosed MS renderer, the data is usually 3-dimensional (3D), so k would normally be equal to 3.

[0244] A balanced tree ensures optimal searches and may be constructed by splitting the data in half at each node by taking the median of the data. However, in typical lightscapes endpoints, at least some light strips are aligned with the x, y or z axis. This means that all elements / actuators of axis-aligned light strips have the same position on one or more axes and that the light positions are not uniformly distributed across the playback environment. This non- uniform distribution means that simply taking the median on alternating axes may result in suboptimal (unbalanced trees).

[0245] Therefore, when configuring the lightscapes Tenderer and building a kd-tree for nearest-neighbor searching (snap-to), the control system may be configured to determine which axis is to be tested at each node by analyzing the data that is being split at that node. In some such examples, the axis that is used to split the data may be the one which:• Has maximal spread, e.g. maximal variance; and• Has the maximal number of unique values.

[0246] Accordingly, the control system may be configured to maximize both the variance and the number of unique values in order to avoid cases in which, for a given axis:• There are many values which are the same with a single outlier, causing large variance; and / or• There are many values which are actually the same or very similar, but due to floating point noise are all appear to be unique.Splitting in either of the foregoing cases is likely to be worse than splitting along a different axis.D24103W001

[0247] As noted above, the tuner may determine that some actuators are not suitable for snapping spatial objects to. In some instances, an actuator may not be suitable for snapping spatial objects to because the actuator has a large area of effect and would “wash out” the spatial object. In some examples, the ternary operator in ALGORITHM 1, “ if e < d,” may be changed to the following:if e < d and j not in unsuitable_actuators[] thenIn the foregoing substituted operator, “unsuitable_actuators[J” represents actuators which are not suitable for spatial object activation. The “unsuitable_actuators[]” may, for example, be a vector of indexes of the actuators which have been identified as being not suitable for spatial object activation.

[0248] Additional metrics for determining which actuator to assign to the DS[x;, y,:,z,-] bin may include, but are not limited to:• Actuator properties such as color gamut, dynamic range and area of effect;• Object properties such as type and shape.

[0249] According to some examples, multiple DSs may be constructed, each optimizing for a particular target. For example, one such DS may be optimal for cuboid objects that are only displaying colors on the black body curve. We can do this by simply augmenting the error function e <- ||P — Los|| of ALGORITHM 1. The augmented error function may be a composite in which multiple costs are combined using Lagrange multipliers. For example, to produce a DS for objects that require a wide color gamut, the augmented error function may be as follows:e «- AP||P - U>os\\2+ (1 - 2P)max(0, 1 - L“C5)In the foregoing expression,represents the Lagrange multiplier used to combine the two cost functions andrepresents the normalised area or volume of the color gamut of the jthlight. This normalization may, for example, be taken with respect to the reference color space used by a color management system.

[0250] However, in some instances the Tenderer may be snapping the spatial object to a small actuator in the middle of an array of actuators. In such instances, simply setting At= 1 would generally provide undesirable actuations. To mitigate this effect, in some examples the Tenderer may set the spatial object’s position to the position of the kth actuator, as follows:D24103W001

[0251] The Tenderer may then re-evaluate the object-actuator activation function across the endpoint.

[0252] In some alternative examples, the DS may return multiple actuator indexes, e.g., as follows:K= DS[xf

[0253] In such instances the Tenderer may set all columns corresponding to indices of the K vector of the ilhrow of Asto be 1, effectively snapping the spatial object to multiple actuators at once. This is potentially useful, because the configuration generator is not bound by computational complexity that is practical at typical frame rates of the Tenderer. As a result, the configuration generator can determine an optimal K vector for a given object position, and potentially also one or more of the following:• Object size;• Object velocity;• Object brightness;• Object color.In some examples, the DS would also be indexed by these additional parameters.

[0254] The K vector may, in some instances, return the indices of optical actuators in the order of decreasing luminance. In some examples, a spatial object may include metadata that indicates the total amount of brightness associated with the spatial object. In such instances, the Tenderer may traverse through the K vector, setting the columns ofto be 1 until the cumulative luminance associated with that spatial object is sufficient according to the metadata.

[0255] Some implementations may involve augmenting the snapping of spatial objects by first computingand then, for all spatial objects which are required to snap to actuators and which do not have at least one actuation value of 1 (in other words, there is at least a single 1 in the objects row of As), performing the snapping process after analyzing the already-computed Asmatrix. Some such examples may involve applying a scaling to the jthcolumn of di j or Dtj when computing the argmin() above to find k. The scaling may be desirable in order to reduce the impact of snapping to actuators already activated by other objects that may be:D24103W001• Higher-priority or on a higher layer, which could cause the snapping of the current spatial object to be useless if the blending and mixing laws would not result in the current spatial object being actuated; or• Lower priority, which could cause causing distortion to the already-rendered scene.

[0256] Some examples may involve using the bed activation matrix AB to mask or augment the search for k when snapping an object to an actuator.Configuring Extended Light Fixtures

[0257] Light fixtures are not literally point sources of light in a playback environment. Some light fixtures may be referred to herein as extended light fixtures, or simply as extended lights. Extended lights, when actuated, may cause one or more areas or regions of a playback environment to be noticeably illuminated. From a viewer’s perspective, the light source is effectively extended across such areas / regions, which may be referred to herein as extended light volumes. Extended light volumes may be used in combination with the sensory objectactuator activation function to control activation of light fixtures. In some instances, if some point within an extended light volume corresponding to a light fixture intersects with a sensory object’s volume, a light fixture corresponding to the extended light volume may be actuated even if the light fixture's position does not intersect with the sensory object’s volume at that time.

[0258] Extended light volumes can be described in various ways, e.g., as parametric volumes such as cuboids, ellipsoids, etc., or as arbitrary shapes constructed from a mesh of vertices. In some implementations, extended light volumes may be automatically calculated at configuration time using playback environment metadata, such as metadata about one or more of the following:The type of light, including but not limited to:o Luminosity;o radiation pattern; and / oro color gamut;The light’s position and orientation within the endpoint;Metadata about the surfaces within the endpoint such as:o Location and shape of surfaces in the endpoint;o Diffusivity of materials making up the surface of the endpoint;D24103W001Metadata about the ambient conditions within the endpoint such as ambient light levels, which are useful in determining perceptual bounds of computed extended lightvolumes;Metadata describing the position of the users in the endpoint: user perspective affects the effective perceptual bounds of computed extended light volumes.

[0259] In some examples, photometric simulations may be run using metadata such as the foregoing in order to estimate the perceptual effect each light has on the playback environment. In some examples, such photometric simulations may be based on information regarding likely user positions and orientations as well as ambient light levels. Such simulations may be used to determine and / or threshold the extended light volume within the playback environment in which a light actuator has a perceptual effect. This extended light volume may be parameterized or tessellated to produce a mesh of vertices describing the extended light volume.Automatically Assigning Extended Light Volumes

[0260] Accordingly, given information regarding the properties of a light fixture, the location and orientation of the light fixture, and in some cases information regarding the layout, furniture, surface reflectivity of walls and object, etc., a control system may be configured to estimate the region in an endpoint that a particular light fixture illuminates sufficiently such that the control system should treat the light fixture as an extended light fixture. This process may involve simulating the photometric response of the room, e.g., from a user’s perspective. The input to such a simulation may include some or all of the following:• The playback environment layout, e.g., information regarding walls, ceiling, floor, desks, furniture, etc.;• The properties of the surfaces in the room, such as their colors, diffuse and specular reflection coefficients, etc.;• Actual or estimated user position(s); and / or• Light fixture color, intensity, position and orientation, as well as the presence or absence of light shades, and light shade properties if present (e.g., color, transmissivity or reflectivity, etc.

[0261] Ignoring the color of the light, a control system may be configured to use a simple Lambertian (diffuse reflection) model to estimate the intensity of the light observed by the user as follows:P = mTn IRD24103W001

[0262] In the foregoing equation:- P represents the intensity of the light reflected at X;- X represents the 3D position in the room (on a surface) where we are evaluating the reflected light intensity;m = Ljos- X represents the anti-incident vector;Losrepresents the 3D position of the light;in =m / ||m|| represents the L2 normalized incident vector;n represents the normal vector of the surface at X;- R represents the reflection coefficient of the surface at X; and1 represents the illuminating light intensity at X. This factor accounts for the intensity of the light emitted by the light fixture, the radiation pattern of the lightfixture (which is evaluated in the direction of — in from lJj0Saccounting for the orientation of the light), and the distance from the light to X.

[0263] In some examples, the control system may be configured to evaluate P at a range of X positions around the playback environment for every light. According to some such examples, if the value of P over an extended region is sufficiently bright then this light is automatically deemed to be an extended light. An “extended region” could be, for example, a region that is larger than a threshold dimension (e.g., 10 cm, 15 cm, 20 cm, etc.) in any direction. One example is a light strip with a single addressable segment that is longer or wider than a threshold, e.g., a single light which may be Im or more longer in one dimension. A light that is “sufficiently bright” is one that results in a perceivable amount of light by the viewer and is dependent on the viewing conditions. In dark conditions, “sufficiently bright” could mean a luminance level as low as 0.05 nits, 0.10 nits, 0.15 nits, etc.

[0264] In more complex scenes, obstructed views from the user position to X should be accounted for when evaluating if the value of P over an extended region within the endpoint is sufficient to convert it to an extended light. For example, the control system may be configured to evaluate information regarding actual or estimated user positions, as well as the locations and dimensions of furniture, interior walls, etc., relative to light locations and orientations, in order to determine whether to determine whether a light should be treated as an extended light.

[0265] More accurate estimates would account for the spectral distribution of the illuminating light source and the spectral reflectance of the surface. However, as here, someD24103W001implementations may implement a simpler model in which a scalar value is deemed sufficient to describe the reflection coefficient.Auto-Assignment of Spatial Object Example

[0266] In one example, an endpoint has multiple extended lights which spatially overlap high-density light strips having a high spatial resolution. In this example, the control system is configured to set the elements of m0corresponding to extended lights that overlap the high-density light strips to zero for when o = spatial object. Here, morepresents the o‘hmasking vector corresponding to the index specified by the object’s metadata.

[0267] Whilst we can implement simulations and algorithms to automatically determine the extended light volume, a tuner may desire to control the process of constructing the volume. Some implementations provide a tuning tool that provides a user one or more of the following:The ability to visualize the performance of the extended light volume in a GUI; The ability to manually define the extended light volume parametrically;The ability to manually define the extended light volume using an arbitrary mesh; - The ability to visualise object trajectories through the endpoint space where the actuators activated by the extended light volumes are shown. This may, for example, involve presenting a GUI with a set of standard object trajectories or test vectors; The ability to visualize the activation of non-extended lights;- The ability to tag actuators as not suitable for snapping spatial objects to; and / or - the ability to manually alter the parameters used to simulate actuators in the endpoint including the metadata listed above.Configuring Auto-assignment of Spatial Sensory Objects to some Actuators and Ambient Objects to Other Actuators (Such As Extended Lights)

[0268] Determining automatic assignment of spatial objects to suitable actuators in the endpoint is similar in nature to constructing masks for a bed-actuator mapping and the DS. Some such examples involve determining and applying a masking mechanism such that AR, I — As,i © m0In the foregoing equation,;represents the i,hrow of the automatically routed actuator activation matrix, As irepresents the i’hrow of the spatially activated actuator activation matrix and m0represents the o'1' masking vector corresponding to the index specified by the object’s metadata.D24103W001

[0269] In some examples, the masking vector m0may contain only Os and Is. Such a masking vector can, for example, be constructed as follows:Setting all elements to 1; andFor all actuators that are not suitable for actuation of spatially activated objects, set the element corresponding to the actuator index to 0.

[0270] Actuators which are not suitable for rendering spatially activated object can include one or more of the following:Actuators which have extended light volumes: rendering spatially activated objects to these actuators may “wash out” the rendered scene;Actuators which have other properties that mean they are unsuitable for rendering spatial objects which could include, but are not limited to:o Poor temporal resolution / update rate of the actuator; and / oro An array of actuators with poor intra-array spatial resolution.

[0271] According to some implementations, a tuning tool may provide at least partial control to a user / tuner over the process of constructing the masking vector mk. such as:The ability to visualize the performance of the masking vector in a GUI, which in some examples may include the ability to visualize the performance of the masking vector for each object type (each masking vector);- The ability to manually define the masking vector, which may involve:o using a masking vector computed automatically as a base; oro using a bed-actuator map as a base;The ability to visualise sensory object trajectories through the endpoint space where the actuators activated by the masking vector are shown, e.g., involving one or more of the following:o Using a GUI with a set of standard sensory object trajectories or test vectors;and / oro Providing one or more types of photometric visualisation;- The ability to run automatic algorithms to construct the masking vector parameterised by, for example, one or more of the following:o Allowable light-volume parameters (shape, size and orientation); o Required temporal resolution;o Required spatial resolution; and / oro Relative position to the user.D24103W001Screen-Anchored Sensory Objects

[0272] A content creator generally creates sensory objects in a creation tool environment using one or more types of sensory content creation tools. The creation tool environment, which also may be referred to herein as a reference environment, has reference environment dimensions. For simplicity, the reference environment may be a virtual rectangular prism, such as a virtual cube in which the reference environment dimensions are equal along the x, y and z axes of the reference environment. The actual playback environment in which the sensory content is rendered will generally not be a cube-shaped environment. In some instances, the playback environment may be a vehicle. Even if the playback environment is a room having the shape of a rectangular prism, the room will typically not be cube-shaped.

[0273] Accordingly, it is often the case that the reference environment shape and the playback environment shape are different. Therefore, unless some type of normalization process is applied, at least some sensory objects that are rendered in the playback environment may have different sizes than the content creator intended.

[0274] A screen-anchored sensory object is a sensory object (such as a light object) which is created in a reference room with a reference screen, where the relative position of the sensory object and the screen are important in conveying the creator’s intent. This may, for example, be because the creator wants a rendered light object to be spatially congruent with content that is displayed on the screen. In some instances, it may be important to retain the spatial relationship of a light object with the screen for functional purposes, e.g., to render some sort of progress bar indicator, directional cue or other visual cue — for example, in a gaming context — where the location of the activated light fixture(s) relative to the screen is a part of the visual cue.

[0275] Figures 9 A and 9B show examples of screen-anchored sensory objects. As with other figures provided herein, the types and numbers of elements shown in Figures 9A and 9B are merely provided by way of examples. Other implementations may include more, fewer and / or different types and / or numbers of elements.

[0276] Figure 9A shows an example of a front wall of a reference room 10000 having a reference TV screen. Figure 9A also shows screen- anchored sensory objects that have been created in the reference room 10000. Figure 9B shows an example of a front wall of an endpoint 10050 (a playback environment) where the sensory objects shown in Figure 9 A are being rendered.

[0277] Figures 9 A and 9B include the following elements:D24103W00110000: A reference room wherein content is created using a creation tool;10005: A front wall of the reference room 10000;10050: A playback environment, which is an endpoint room in this example;10055: A front wall of the endpoint room 10050;20000: A reference TV;20100: A light object created in the reference room 10000;20101: A light object created in the reference room 10000;20102: A light object created in the reference room 10000;20010, 20011, 20012, 20013, 20014, 20015, 20016 and 20017: spatial segments of the front wall 10005;20020, 20021, 20022, 20023, 20024, 20025, 20026 and 20027: spatial segments of the front wall 10055 of the endpoint room 10050;20020: The TV in the endpoint room 10050;20200: The light object 20100 as rendered in the endpoint room 10050;20201: The light object 20101 as rendered in the endpoint room 10050; and 20202: The light object 20102 as rendered in the endpoint room 10050.

[0278] According to some examples, the size and position of the reference TV 20000, along with the size of the reference room 10000, is transmitted as metadata as part of the sensory content received by a sensory Tenderer of the endpoint room 10050. In order to retain the relative position of sensory objects rendered in the endpoint room 10050 with the actual TV screen 20020 in the endpoint room 10050, a control system implementing the Tenderer has scaled the positions of the light objects 20100, 20101 and 20102 according to the relative positions of the reference TV 20000 and the actual TV 20020 after a normalization process. This scaling process may, for example, involve a piece-wise linear interpolation where the wall segments map as follows:• 20010 maps to 20020;• 20011 maps to 20021;• 20012 maps to 20022;• 20013 maps to 20023;• 20014 maps to 20024;• 20015 maps to 20025;• 20016 maps to 20026; and• 20017 maps to 20027.D24103W001

[0279] According to this example, the control system determines which segment or segments the sensory object is in in the reference domain. For example, light object 20102 is in segment 20015 of the reference room 10000, which is defined by the position and size of the reference TV 20000. Similarly, the segment 20025 is defined by the position and size of the actual TV 20020 of the endpoint room 10050. In this example, the control system determines the position of the light object 20202 in the endpoint room 10050 — which is the rendered version of light object 20102 of the reference room 10000 — by interpolating between the bounds of the segment 20025 according to the position of the object in the reference room 10000.

[0280] It may be observed that the width (y dimension) and height (z dimension) of the reference room 10000 are different from the width and height of the endpoint room 10050: in these examples, the height of the endpoint room 10050 is less than that of the reference room 10000, whereas the width of the endpoint room 10050 is greater than that of the reference room 10000. Therefore, as a result of the above-described mapping process, the light objects 20100, 20101 and 20102 — which are intended to have equal height and width dimensions in this example — are rendered with distorted sizes and shapes in the endpoint room 10050. Some disclosed examples describe normalization processes that can alleviate this type of sensory object size and shape distortion.Aspect-Preserving Screen Anchoring Projections

[0281] Figures 10, 11, 12A and 12B show examples of aspect-preserving screen anchoring projections. As with other figures provided herein, the types and numbers of elements shown in Figures 10-12B are merely provided by way of examples. Other implementations may include more, fewer and / or different types and / or numbers of elements.

[0282] As noted above, the projection illustrated in Figures 9 A and 9B distorts the rendered sensory object scene to fit the aspect ratio of the front wall 10055 and the size and location of the TV 20020. To mitigate the effects of a change in aspect ratio of the front wall, in some examples the control system may be configured to resize the received sensory content to fit the endpoint front wall dimensions. Some such examples may involve determining a single scalar normalizing factor that is applied to both the y and z dimensions.

[0283] In Figure 10, the height of the front wall 10055 is used to normalise the spatial object sizes and positions in the endpoint 10050. The endpoint 10050 in Figure 10 is wider than it is high. According to this example, there is a segment 21000 in which no points map from the reference room 10000. Accordingly, in this example, including the segment 21000 creates a “dead space” in the endpoint 10050, in which there will be no actuation of lights or otherD24103W001actuators. In this particular example, the resized spatial segments of the front wall 10055 are based on the smaller of the two sides of the TV 20020.

[0284] Figure 11 shows an example of resizing the spatial segments of the front wall 10055 according to the larger of the two sides of the TV 20020 and of preserving the relative sizes of the projected spatial segments that surround the TV 20020. As a result of applying this method, some spatial segments are cropped and some positions in the reference space do not map to a physical point physically in the endpoint 10050.

[0285] In the example shown in Figure 11, spatial segments 23020, 23027, 23026, 23025 and 23024 map to points not physically in the endpoint 10050 and therefore sensory content that is mapped there will not be reproduced / played back in the end point 10050. The projections of spatial segments from the reference room 10000 to the endpoint room 10050 can be summarised as follows:• Spatial segment 20010 of the reference room 10000 is projected to spatial segments 23120 and 23020 of the endpoint room 10050;• Spatial segment 20017 is projected to spatial segments 23127and 23027;• Spatial segment 20016 is projected to spatial segments 23126 and 23026;• Spatial segment 20015 is projected to spatial segments 3125 and 23025; and• Spatial segment 20024 is projected to spatial segments 23124 and 23024.Resize and Fold

[0286] Figures 12A and 12B show examples of a “resize and fold” aspect-preserving and screen-anchoring method. In this approach, the front wall 10055 of the endpoint room 10050 is normalized in such a way that the aspect ratio of the front region of the reference room 10000 is maintained. Instead of spatial segment 24000 of the endpoint room 10050 being dead space, as in the example shown in Figure 10, the content on the right wall 24101 of the reference room 10000 is mapped onto spatial segment 24000. Accordingly, the spatial segment 24000 may be thought of as a warped space in which spatial sensory objects located on the right wall in the reference room 10000 are mapped onto the front wall 10055 in the endpoint 10050 in order to provide continuity in the spatial sensory content. Such methods may be performed in various ways, including the following:• Projecting the coordinates using a pseudo-hemispherical system where the front wall 10050 is the plane of the projected hemisphere; orD24103W001• Aliasing the spatial sensory object coordinates so that spatial sensory objects can appear on the right wall and the front at the same time.General Approaches to Screen-Anchored Rendering

[0287] In general, there are a few trade-offs to be made when projecting spatial sensory objects into normalized room coordinates for the purpose of screen- anchored rendering that include:• Maintenance of aspect ratio versus distorting rendered spatial sensory objects’ shapes and sizes;• Clipping / cropping of content that gets projected out of the physical space versus not clipping or cropping the content.

[0288] Such trade-offs can be mitigated according to the desired behaviour indicated by the creator by various combinations of the following:• Resizing the content;• Clipping / cropping the content;• Object size normalisation; and / or• Warping the coordinates.

[0289] Furthermore, methods which produce a dead zone, such as that shown in Figure 10, can be augmented by:• When rendering light objects, computing a wash to be rendered to the dead zone based on the light objects rendered elsewhere in the playback environment. This method will fill in the dead zone with content and colors that are congruent with the other portions of the rendered lightscape;• Repeating / aliasing of the light objects from the adjacent walls and segments onto the front wall of the playback environment.Configurations Involving Assumed User Positions and Multiple- Screen Endpoints

[0290] Some example configurations involving assumed user positions and multiple-screen playback environments are described in this section with reference to Figures 13, 14, 15 A, 15B and 16. As with other figures provided herein, the types, numbers and arrangements of elements shown in Figures 13-16 are merely provided by way of example. Other implementations may include more, fewer and / or different types and numbers of elements.D24103W001

[0291] Figure 13 shows a top-down view of a reference room in which content is created with respect to a reference TV and a reference user position. According to this example, Figure 13 shows the following:• 10000: A reference room wherein content is created using a creation tool;• A front wall of the reference room 10000, viewed from the top (along the z axis); • 20000: A reference TV• 30002: A reference user position;• 30001: A distance from the reference TV 20000 to the reference user position 30002, which is a distance along the x axis in this example.

[0292] Figure 14 depicts a top-down view of a playback environment with multiple screens and users. In this example, Figure 14 shows the following:• 30500: An endpoint with multiple screens and users;• 30020: Screen 1• 30200: User 1:• 30101, 30102, 30103 and 30105: light strips which are designated as belonging to the screen 1 domain;• 30106 and 30107: lights which are designated as belonging to the screen 1 domain;• 30021: Screen 2;• 30201: User 2;• 30111 and 30112: light strips which are designated as belonging to the screen 2 domain.

[0293] The light strips 30101-30107 are associated the screen 1 domain for the purposes of rendering screen- anchored light objects corresponding to content displayed on screen 1. The light strips 30111 and 30112 are associated with the screen 2 domain for the purposes of rendering screen- anchored light objects corresponding to content displayed on screen 2.

[0294] In this section we will utilise the scaling of room coordinates in the XZ dimension (the plane of the front wall) described at the start of the previous section. This stretches the coordinates of the lights to fit the aspect ratio of the room when normalizing. Any of the XZ projections and techniques disclosed in the previous section can be applied here.

[0295] Figures 15 A, 15B and 16 show top-down views of reference rooms. Figures 15 A and 15B show how the light strip positions and other light positions shown in Figure 14 can be mapped to a reference room for screen- anchored objects. In these examples, Figures 15A and 15B show the following:D24103W001• 10000: a reference room;• 30020: Screen 1;• 30101, 30102, 30103 and 30105: light strips which are designated as belonging to the screen 1 domain;• 30106 and 30107: lights which are designated as belonging to the screen 1 domain;• 30021: Screen 2;• 30111 and 30112: light strips which are designated as belonging to the screen 2 domain;• 30600: Screen domain 1 mapped onto the reference room 10000; and• 30601: Screen domain 2 mapped onto the reference room 10000.

[0296] Figure 16 shows how the light strip positions and other light positions shown in Figure 14 can be mapped to a reference room for room-anchored objects. In this example, Figure 16 shows the following:• 10000: a reference room;• A reference TV;• 30101, 30102, 30103 and 30105: light strips which are designated as belonging to the screen 1 domain;• 30106 and 30107: lights which are designated as belonging to the screen 1 domain; and • 30111 and 30112: light strips which are designated as belonging to the screen 2 domain.Screen-Domain Content Mapping

[0297] In this section, methods are described for flexibly rendering sensory content for multiple screen domains created without knowledge of the endpoint. Such methods may be implemented by a control system, which may be a control system to implement an MS renderer or a lightscape Tenderer. These methods use source IDs, logical IDs and physical IDs to abstract the content (source) from the endpoint (physical).

[0298] The source ID is an identifier attached to a sensory object that indicates which content screen source ID a sensory object is associated with. A content creator may, for example, use the source ID to associate sensory objects with multiple video streams that are rendered into an endpoint as a part of a multi-sensory experience involving lightscapes. Specifically, a content creator may create a multi-sensory experience with multiple video feeds. The content creator may associate screen-anchored sensory objects to each of these video feeds. As a result, the source ID of the lightscapes’ screen-domain objects becomes analogous to the ID of the video feed amongst the multiple video feeds.D24103W001

[0299] The logical ID is the intermediate domain which abstracts the creation from the rendering and provides flexibility. For a given lightscapes metadata specification, a number of logical domains may be specified (for example, 4, 6, 8, 10, 12, etc.). In some examples, when a content creator creates screen-domain sensory content, the content creator provides a map from all source IDs (which may correspond with the number of video feeds in the sensory content) to the entire logical domain range regardless of the number of content source IDs. In one example, if the lightscapes metadata specification contains 8 logical domains, and the sensory content only contains 2 screen domains, the content creator must provide a mapping for each of the logical IDs from the 2 source IDs.

[0300] The physical IDs are specific to an endpoint. For example, an endpoint such as a vehicle may have 1 display screen in the front and 2 display screens in the headrests of the front seats, configured to be viewed by passengers in the backseat. This example endpoint has 3 physical display screen domains (physical IDs), one corresponding to each display screen. In some examples, during configuration of the Tenderer, the tuner will decide to which logical domain each of these physical display screen domains maps.Example Screen source-> Logical-> Physical Mapping Scenario

[0301] For the automotive example given in the previous paragraph in which there are 3 display screens, the tuner may, for example, select one of the following options:• Physical option A: Set the physical ID associated with the display screen in the front of the car to be mapped to logical domain 1 and the two physical IDs associated with the display screens embedded into the front seats to be mapped to logical domain 2; or• Physical option B: Set the physical ID associated with the display screen in the front dash of the car to be mapped to logical domain 1 and the two physical IDs associated with the display screens embedded into the front seats to be mapped to logical domain 2 and logical domain 3 respectively.

[0302] In some implementations, for an endpoint configuration, this physical to logical ID mapping may be fixed. However, in some examples, there may be multiple physical to logical ID mapping configurations which can be selected by the user at runtime. In some such examples, there may be a limited static set of these physical to logical ID mapping configurations. According to some examples, when the user switches between these physical to logical ID mapping configurations, this action will cause the video content is being played in each of the screens to be switched. That is, this switching action may be a mechanism forD24103W001view selection in a multi-screen multi-view (content) experience. Depending on the content, if a dynamic source ID to logical ID mapping is present, a dynamic mapping of source ID to physical ID screen domains may occur. The content creator may choose this dynamic mapping option. In some examples, the dynamic mapping may vary between pieces of content.

[0303] Figures 17A, 17B, 17C, 17D, 17E, 17F. 17G, 17H, 171. 17, 17K, 17L. 17M, 17N and 170 show examples of mapping source IDs to physical IDs and logical IDs. In these example, we illustrate a system with 4 logical domains and four corresponding logical IDs (0, 1, 2 and 3). Although some systems already developed by the present inventors will support up to 16 screen IDs, and future implementations may provide more than 16 screen IDs, we have limited the number of logical domains to 4 in these examples for clarity and simplicity. As with other figures provided herein, the types and numbers of elements shown in Figures 17A-17O are merely provided by way of example. Other implementations may include more, fewer and / or different types and numbers of elements.

[0304] In the examples shown in Figures 17A-17C, the content includes four video feeds with associated screen-domain lightscape content and four corresponding source IDs (0, 1, 2 and 3). In this example, the content creator has a maximal number of video feeds (due to the limit of 4 logical IDs) and — as shown in Figure 17A — provides a 1:1 mapping of source ID to logical ID in the content metadata. According to these examples, the endpoint has three display screens and three physical screen domains. Here, physical option A, above, is chosen as the physical ID to logical ID mapping in the endpoint configuration, which results in the endpoint configuration mapping shown in Figure 17B. The corresponding mapping of source ID to physical ID that is applied at run time is shown in Figure 17C.

[0305] In the examples shown in Figures 17D-17F, the content includes two video feeds with associated screen-domain lightscape content and two corresponding source IDs (0 and 1). Figure 17D shows an example of the mapping the two source IDs to the four logical IDs in the content metadata. According to these examples, the endpoint has three display screens and three physical screen domains. Here, physical option B, above, is chosen as the physical ID to logical ID mapping in the endpoint configuration, which results in the endpoint configuration mapping shown in Figure 17E. The corresponding mapping of source ID to physical ID that is applied at run time is shown in Figure 17F.

[0306] In the examples shown in Figures 17G-17I, the content includes three video feeds with associated screen-domain lightscape content and three corresponding source IDs (0, 1 and 2). Figure 17G shows an example of the mapping the three source IDs to the four logical IDs inD24103W001the content metadata. According to these examples, the endpoint has three display screens and three physical screen domains. Here, physical option B, above, is chosen as the physical ID to logical ID mapping in the endpoint configuration, which results in the endpoint configuration mapping shown in Figure 17H. The corresponding mapping of source ID to physical ID that is applied at run time is shown in Figure 171.

[0307] In the examples shown in Figures 17J-17L, the content includes one video feed with associated screen-domain lightscape content and one corresponding source ID. Figure 17J shows the single source ID mapped to the four logical IDs in the content metadata. According to these examples, the endpoint has three display screens and three physical screen domains. Here, physical option B, above, is chosen as the physical ID to logical ID mapping in the endpoint configuration, which results in the endpoint configuration mapping shown in Figure 17K. The corresponding mapping of source ID to physical ID that is applied at run time is shown in Figure 17L.

[0308] In the examples shown in Figures 17M-17O, the content includes four video feeds with associated screen-domain lightscape content and four corresponding source IDs. Figure 17M shows the four source IDs mapped to the four logical IDs in the content metadata. According to these examples, the endpoint has two display screens and two physical screen domains. The endpoint configuration mapping of this example is shown in Figure 17K. The corresponding mapping of source ID to physical ID that is applied at run time is shown in Figure 170.Rendering Volumetric Fields

[0309] The term “volumetric field” as used herein refers so a field within a volume, the field being defined by functions. The functions may, for example, be parameterizable by a content creator. Volumetric fields may or may not be spatially uniform, depending on the particular implementation. In some examples, a content creator may specify in the content metadata whether a volumetric field is spatially uniform. Functions which may be parameterizable across the volumetric field include, but are not limited to, the following:• Spatially modulated basis functions, such as sinusoidal functions that are a function of the spatial position across the field;• Temporally modulated basis functions,, such as sinusoidal functions that are a function of time;• Signal mixing functions;D24103W001• Signal multiplication functions;• Activation functions, such as:o Non-linear transformations e.g. a = x2; oro Non-linear thresholding, e.g., a = min(x, b); and / or• Deterministic generators parameterised across the volume, e.g., pseudo-random noise.

[0310] The volumetric field may be sampled at spatial intervals, which may be either uniform — that is, sampled at regular intervals across the space of the endpoint — or non-uniform. Non-uniform sampling may, for example, be based on information in the content metadata. In one such example, information in the content metadata may indicate, e.g., sampling centred at a display screen, such as a television (TV) display screen. In some examples, the sampling may be log-space sampling. According to some examples, the volumetric field may be sampled temporally each time the Tenderer runs a process call.

[0311] Volumetric fields, volumetric textures, or both, can be used to augment and / or modulate the properties of spatial objects across an endpoint. For example, a volumetric field may be created in which the amplitude is spatially and temporally modulated. This amplitude may then be applied to any spatial objects in the endpoint and only then will the spatial and temporal modulations affect the actual actuators. One example may involve “flicker” modulating of all light objects in a room.

[0312] The output of a volumetric field can be a scalar or a vector. In some examples, the output of a volumetric field may be used to directly affect an actuator. For example, a volumetric field may produce an RGB color tuple, which may be directly applied to one or more actuators. In some examples, the output of a volumetric field may be used to modulate one or more actuators. For example, a scalar volumetric field may modulate the amplitude of the actuators in a playback environment.

[0313] According to some examples, the Tenderer may generate a parametric volumetric field tensor VPG IPf'v-';>< Nyx Nz*NP, where Nx, Nyand Nzrepresent the numbers of spatial samples in the x, y and z axes of the room. These numbers are equal when the spatial sampling is uniform and the room is isotropic. Nprepresents the dimensionality of the volumetric field. In this example, Np= 1 if the volumetric field is a scalar field and Np> 1 if the volumetric field is a vector field.

[0314] If the volumetric field is a scalar field and will modulate the amplitude of the actuators in the endpoint, then the volumetric field may be implemented as follows:A' = As© VPD24103W001

[0315] In the foregoing expression, O represents Hadamard multiplication and A’ represents the modified AAM resulting from the implementation of the volumetric field.

[0316] One way to realize a volumetric field is to specify (a) a spatial modulation function parameterised by the volumetric field’ s origin and the density of the volumetric field and (b) a temporal modulation function. In some examples, the temporal modulation function may be parameterised by a time-frequency relationship and a thresholding function. One such volumetric field that outputs a vector may be expressed as follows:< ) = nwd px+ (f)y+ cpz, 2n~)In the foregoing expression, mod a, b) represents a modulo function which returns the remainder of a divided by b, and < >x, py, (f>zrepresent the spatial phases associated with each of the dimensions of a playback environment at the position (x,y,z). In some examples, < >x, < >y, < / )zcan be computed as follows:<Px = (x - + ftfy = (y “ °y°Sfy + ftfz = (z - O^os)fz+ (pt

[0317] In the foregoing expression, fx,fy,fzrepresent the spatial frequencies associated with the x,y,z dimensions, respectively and < >trepresents the phase of the temporal modulator, which may be computed as follows:< >t = t ft

[0318] In the foregoing expression, t represents the current time and ftrepresents a timefrequency temporal modulation parameter. The corresponding volumetric field tensor VPmay be expressed as follows:T / > f c if (p < ap,(x,y,z) [Os otherwise

[0319] In the foregoing expression, a represents a threshold between zero and 2TI and is a parameter that a content creator may control, c represents a codeword vector and 0s represents a vector of zeros having the same length as c. For a lightscape modality, c could be an (R, G, B) codeword, for example.

[0320] According to some examples, c itself could be generated in various ways, such as:• as a function of spatial position;D24103W001• as a function of some other parameter such as time, or of a parameter interactively set by a user;- by a random generator function, etc.

[0321] A content creator may choose to have some spatial dimensions not dependent on the origin of a volumetric field. For example, one could set < >zas follows:< Z»z = (z - 0.5) / z+ < Z>t

[0322] In order to compute FP(x y Z) one may choose to substitute the positions of the actuators for the (x,y,z) locations. Alternatively, one may choose to only evaluate the volumetric field at a grid of spatial locations that are independent of the actuator locations and then perform some interpolation or activation.

[0323] Some examples may involve using precomputed data obtained at Tenderer configuration / tuning / initialization time to bound or scale the parameterized functions used to realize a volumetric field. For example, one may use such precomputed data to limit the range of amplitude modulation according to knowledge of the actuator capabilities, to scale the spatial-frequencies according to an endpoint room scale, etc.

[0324] Some examples may involve limiting the range of an amplitude-modulated scalar volumetric field that may simply be a temporally modulated field, for example as follows:^P,(x,y,z) 1

[0325] Some examples may involve augmenting the foregoing expression as follows:^P,(x,y,z) ®

[0326] In the foregoing expression, a represents a modulation depth limiting factor that will avoid actuators being continually driven to actuation amplitudes below a when one sets A' = AsO VP. There are many ways in which one can augment the modulation depth, for example as follows:^p. Cx.y.z) = 1 - « (1 + sin(( / >t))The foregoing expression leaves the modulation result balanced.

[0327] A volumetric field can provide various types of functionality, including but not limited to the following:• Augmenting the values of Asdirectly if the field is scalar as shown earlier;• Augmenting the object-actuator activation functions directly or indirectly, for example:D24103W001o By modulating the distance matrices, e.g. Ei:]- — D:jVPfor Np- 3 volumetric fields or Eii7= dt Vp for scalar volumetric fields;o By modulating spatial object sizes;o By modulating spatial object positions;o By modulating actuator positions;• Augmenting the codewords, e.g., for lightscapes the R channel of an RGB codeword may be modulated by a scalar volumetric field; and / or• Producing a vector field of codewords which will be mixed in with the remainder of the objects being rendered.

[0328] If a vector volumetric field produces codewords and activation values for each actuator in the endpoint, then we have Avwhich is a Nf by Namatrix in which N represents the number of volumetric fields being rendered and each row of Avrepresents the volumetric field activation for a particular field for every volumetric field being rendered (there may be multiple volumetric fields in the content stream being rendered). Then, we can mix the volumetric field in by

[0329] Bed-based content may also be amalgamated as follows:Rendering Volumetric Textures

[0330] Volumetric textures are similar to volumetric fields in the sense that they can provide either a scalar or vector field over a spatial coordinate domain of the endpoint. The main difference between volumetric textures and volumetric fields is that the field corresponding to a volumetric texture is derived from data, not by functional parameters provided by the creator. For example, a content creator may choose to paint a 2D or 3D image and use this as the basis for the volumetric field. A greyscale image could produce a scalar volumetric field. The content creator can take this asset and determine how it will be sampled to produce the volumetric texture. If the image is 2D, for example, the content creator may choose to place this asset on the XY plane (parallel to the floor of the playback environment) and may choose to make the field constant along the Z axis.D24103W001

[0331] Volumetric textures may be combined with volumetric fields to create desired effects. For example, a scalar volumetric texture may be used to define some arbitrary function over the spatial domain which drives time-frequency oscillators at particular spatial locations.

[0332] Volumetric textures can be used in the same way that volumetric fields are used to augment the actuator activation matrix and object activation functions, e.g., as described in the previous section.

[0333] Because volumetric textures are not defined in terms of parameterizable functions, it is not possible to evaluate them at arbitrary (x,y,z) spatial locations. Thus, we cannot evaluate them at the actuator positions, and we are only able to perform some sort of interpolation or resampling of the volumetric texture if we want to directly project it onto actuators to compute Avor Vp. Evaluating the volumetric texture at arbitrary (x,y,z) spatial locations to compute Avor Vp. may, for example, involve:• Nearest-neighbour sampling;• Trilinear interpolation; or• Tricubic interpolation.

[0334] If the actuator positions are fixed in the endpoint, precomputed coefficients or other values may be used to speed up the interpolation process. If the volumetric texture is static, the interpolated values may be precomputed once and stored for subsequent use.Interpolated values also may be precomputed once and stored for subsequent use if the volumetric texture is not static, but periodic and the total amount of data is not too large to store.Configuring Volumetric Fields and Textures

[0335] Room-scale normalization for volumetric fields involves scaling the parameters of the volumetric field objects according to the playback environment size (e.g., the endpoint room size). Room-scale normalization may affect properties including, but not limited to spatial frequency, temporal frequency and / or density.

[0336] For example, we may scale a spatial frequency byfx = Pxfxfy = Pyfyfz = PzfzD24103W001

[0337] In the foregoing equations, fx, fyand fzrepresent the normalized spatial frequencies, fx, fyand fzrepresent the spatial frequencies associated with the volumetric field, and px. pyand (3Zrepresent scaling parameters.

[0338] There are many ways to determine the scaling parameters fx. f>yand / ?zincluding, but not limited to, the following:Determining the scaling parameters based on only the bounding box containing the entire playback environment;- Determining the scaling parameters based on only a bounding box containing actuators rendering the experience in the playback environment;Determining the scaling parameters based on being anchored or aligned along at least one axis with a display screen in the playback environment, if one is present;Determining the scaling parameters based on a bounding box containing only a subset of the actuators of the playback environment; and / orDetermining the scaling parameters based on only the bounding box containing only a subset of actuators in the room which are referenced to a particular screen in the playback environment.

[0339] In one example, a room-anchored volumetric field may be scaled based on the playback room size Rslze, as follows:nsizeB - - - - max (Rsize)nsize / ? =y.ymax (7?slze)gsizeB = - - -'zmax (Rsize)

[0340] Volumetric textures should be interpolated in order to determine their relevant value associated with each actuator in the playback environment / room. This is because volumetric textures are created on a sampling lattice defined by the creator which may not coincide with the actuators’ actual positions.

[0341] It is also possible to evaluate a volumetric field where it can be considered to generate a texture on some lattice by the rcndcrcr which is defined by the parameters of the sensory object that the creator has chosen. For this purpose, techniques such as the following may be used to perform the relevant resampling:D24103W001• Nearest-neighbour sampling;• Trilinear interpolation;• Tricubic interpolation.

[0342] If the control system is performing interpolation, then the control system requires coefficients, computed from the position of the actuator relative to the lattice that the volumetric data is defined on, to combine multiple values across this lattice. In some examples, a set of standard lattices that the creator can work with may be defined. If the location of the actuators in the playback environment is fixed, these coefficient may be precomputed as part of a Tenderer configuration or Tenderer tuning process.

[0343] Some examples involve computing coefficients, based on knowledge of the actuators in the playback environment, to regularize the room normalization coefficients even further. For example, some implementations may involve scaling the parameterized spatial frequency by the computed coefficients, e.g., as follows:( nsize r- \max z (RosilzeA) ’ 2 oL irXes J I( nnysize r py\ \max r \Rssilzzee\) ’ 2 'iL iryes J I( RysizeFz\Bz= max I - - — ■. — -, -, z\ max (7?slze) 2LzesJ

[0344] In the foregoing equations, Fx, Fyand Fzrepresent reference frequencies associated with the x,y,z axes of the endpoint / playback environment, and Lrxes, Lrxsand Lrxsindicate the representative spatial sampling provided by the actuators across the x,y,z axes of the endpoint. By taking into account this representative spatial sampling, the control system may ensure that Nyquist sampling is satisfied across the endpoint. In some examples of a control system that is configured to automatically compute the Tenderer configuration, some actuators may be masked out from rendering volumetric fields and textures. In some instances, this may be because the actuators’ spatial resolution is lower than a suitable representative spatial sampling.

[0345] In some examples of a control system that is configured to automatically compute the Tenderer configuration, actuation capabilities — such as dynamic range, quantization loss and luminosity, for example — may be used by the control system to produce additional scaling factors that can be applied to cause one or more of the following results:D24103W001• Limiting the dynamic range of amplitude modulation;• Limiting the spatial frequency of the volumetric data; and / or• Limit the temporal frequency of volumetric data due to update rate limitations of actuators.

[0346] In some instances, a tuner may desire to have control over at least some aspects of the process of computing the factors for scaling volumetric field parameters, volumetric texture parameters, or both. Accordingly, some tuning tool implementations may provide the user / tuner with one of more of the following:1. The ability to visualize the performance of the volumetric objects with and without scaling parameters defined in this section in a GUI;2. The ability to manually define how the volumetric field and / or volumetric texture scaling factors are automatically computed (e.g. enclosing the room vs enclosing the actuators);3. The ability to manually define the volumetric field and / or volumetric texture scaling factors;4. The ability to visualize volumetric object trajectories through the endpoint space where the actuators are activated with or without using scaling factors computed for the playback environment, e.g., via a GUI that shows a set of sensory object trajectories, sensory object test vectors, etc.;5. The ability to manually define lattices for precomputed interpolation coefficients to be determined; and / or6. The ability to manually define actuators which are masked out and prevented from rendering spatial sensory objects.Configuring Room-Scale Normalization and Screen-Domain Normalization

[0347] In some examples, room-scale normalization involves scaling sensory object sizes, Oflze, by the same factor used to normalize the actuator positions. This normalization process may be done so that sensory objects are not physically different sizes in an endpoint room if the sensory objects are the same size in the reference room. Although the term “room” is used for convenience herein, the methods described in this section and elsewhere in this disclosure also apply to non-room playback environments, such as vehicles. The normalization of the fhlight position can be expressed as follows:^position rjvosition^=J LnJ[Q7 t JL JD24103W001

[0348] In the foregoing equation, RslzeG R3represents a size vector indicating the size of the playback environment, e.g., the size of an endpoint room. This size vector may, in some implementations, be computed automatically at configuration time. The size vector may, in some examples:Enclose an entire room in which the playback environment is located;Enclose only the actuators rendering the experience in the playback environment; Be anchored or aligned along at least one axis with the screen in the playback environment if one is present;Enclose only a subset actuators of the room; and / orEnclose only a subset of actuators in the room which are referenced to a particular screen in the playback environment.

[0349] More complex scaling or warping of coordinates may, in some examples, be used for the purposes of rendering screen-anchored sensory objects and room-anchored sensory objects, e.g., for determining how the spatial objects in the endpoint are scaled. Screen-anchored objects are created with the intent of being rendered in the endpoint with a spatial location relative to a display screen. In this case, spatial coordinates may be stretched / warped in the plane of the screen, etc., e.g., as described elsewhere herein. Some disclosed implementations involve rendering screen-anchored objects to multiple screen domains.

[0350] The scaling and offsets used to apply the stretching & resizing of screen-anchored sensory objects and room-anchored sensory objects may be computed automatically by a control system, given:• The size of the playback environment;• the positions of the actuators within the playback environment;• the size and location of the scrccn(s) within the playback environment; and • the assumed location(s) of one or more users within the playback environment. As noted elsewhere herein, trade-offs may be made when performing the stretching and resizing.

[0351] A tuner may desire control over at least part of the process of computing the factors for room-scale normalization, screen-domain rendering, etc. Some tuning tool implementations may be configured to provide the user / tuner with one or more of the following:• The ability to visualize the performance of the scaling of the room in a GUI;D24103W001• The ability to manually define how the room scaling normalization factor is automatically computed (e.g. enclosing the room vs enclosing the actuators);• The ability to manually define the room scale normalization factors;• The ability to visualize object trajectories through the endpoint space where the actuators are activated using room scale normalization;o This may involve providing a GUI and a set of sensory object trajectories and / or test vectors;o This may involve visualizing the activation (e.g., of light fixtures) with and without room scale normalization;• The ability to choose the method in which the room- and screen- domain position scaling is automatically computed (e.g. resize and fold vs. stretch);• The ability to visualize the effective screen domain coordinates of one or more screen domains;• The ability to choose the method with which the screen domain scaling is performed;• The ability to manually determine which screen domain any actuator belongs to;and / or• The ability to manually augment the effective positions of actuators in either the room or screen domain.Actuator-Density-Based Lux Normalization

[0352] In addition to ensuring the perceived color of the actuators is consistent across the actuators, it is desirable that the Tenderer should ensure that the perceived brightness is consistent across the actuators in the playback environment. Actuator-density-based lux normalisation refers to the process of scaling the brightness of the light actuators such that the lux (total emitted light per unit area) is consistent across the playback environment. For example, consider two equal-length light strips on opposing sides of the playback environment. In this example, both light strips have the same physical actuators (LEDs, drivers, circuitry, etc.), such that the same level of actuation of a single element on both light strips results in the same physical actuation in the playback environment. If one of the light strips has twice as many elements, then commanding both light strips to set all of their elements to the same codeword will result in the lights trip with twice as many elements producing twice as much lux and being perceived by the user as being logarithmically brighter.D24103W001

[0353] If the goal is to simply set endpoint-wide light setpoints to the actuators, then in some examples the control system — such as a control system implementing the lightscape Tenderer 501 of Figure 5 or Figure 7 — may be configured to scale the brightness of the j‘hactuator by a brightness scaling factor Bj, which may be computed at Tenderer configuration, tuning or initialisation time. The brightness of an actuator may be scaled as follows:Bout = min (1, Bj Bin)

[0354] In the foregoing expression, the min() operator ensures that for actuators that have Bj > 1, we do not exceed the range of valid codewords. The brightness may be scaled using any suitable color model. In some implementations, a control system — such as the control system implementing the lightscape Tenderer 501 of Figure 5 or Figure 7 — may be configured to compute the brightness scaling factor Bj by analyzing one or more of the following:• The density of the actuator elements within the playback environment:• The radiation pattern and falloff of actuator elements within the playback environment;• The estimated reflections of actuator elements within the playback environment towards assumed or actual user position(s), preferably including the effect of the estimated reflections from materials making up the playback environment and objects within it; and / or• The assumed or actual user position(s).

[0355] To compute Bj the control system may require some reference lux to normalize against. The choice of this reference lux could be, for example:The minimum lux emitted by a single actuator of the playback environment;- The minimum lux emitted by two elements of an actuator array of the playback environment; orA reference value, such as a reference value provided by the content metadata data or a configuration parameter.

[0356] In some circumstances, a subset of actuators of the playback environment may have a significantly lower brightness capability than the other actuators of the playback environment. It may be desirable to ignore this subset of actuators when determining the reference lux to normalize the actuators against. In some such examples, the control system may be configured to set Bj to a number larger than 1 for all actuators that have been ignored when determiningD24103W001the reference lux. This will result in the actuators that have been ignored being boosted in brightness at render time.

[0357] In some instances, the i,hlight object may have an associated target lux value, which may be indicated via metadata. In such cases, the control system cannot compute BLj — which represents the scaling for a particular object- actuator at configuration time — because this data will not be available at configuration time. In some such examples, the control system implementing the lightscape Tenderer may be configured to compute B, in some examples by using Bj as a reference. The computation of Bi may be based, at least in part, on one or more of the following parameters:• The number of actuator elements the light object is being rendered across; and / or • The target lux value.

[0358] Given these parameters, the Tenderer may also augment the position of the light object — in other words, alter the light object / light fixture distance to make the light object seem closer to the light fixture — in order to ensure sufficient actuation (lux) is achieved.

[0359] A tuner may desire to control at least some aspects of configuring how to normalize lux in the Tenderer. Some tuning tools may provide a tuner with one or more of the following:• The ability to visualize the performance of the lux normalization in a GUI. For example, the tuning tool may provide the tuner with the ability to visualize the perfomrance of the lux normalization for a range of different target object lux values and / or for a range of different actuator element activations (e.g., according to light object position and size);• The ability to manually select particular actuator arrays or elements to be ignored when computing the reference lux;• The ability to manually apply radiation models to actuators; and / or• The ability to manually apply playback environment reflection / scattering models to actuators.Rendering Images and Videos

[0360] Images and videos differ in nature to volumetric textures (and fields) in the sense that images and videos are generally not evaluated or sampled across the spatial domain of an endpoint. Instead, images and videos are generally rendered to one or more actuators comprising an array of actuators that provide sufficient resolution and coverage to render anD24103W001image or video. The reader will note that, as used herein, the terms “images” and “videos” are not necessarily optical images and videos. For example, an image may simply be a 2D or 3D signal with regular uniform spatial sampling. One example of a non-optical image is a haptic signal that is defined across an array of spatial points that could, for example, be rendered to an array of actuators on a vest. A video is simply an image in which the signal changes over time.

[0361] A content creator may import an image or video into their sensory creation tooling and associate the image or video with a sensory object. In some instances, a content creator may create a sensory object corresponding with an image or video. The position and size of this sensory object can determine the bounding region in which the sensory object is able to be rendered upon a suitable array of actuators.

[0362] A content creator may also determine the way in which an image or video is rendered upon an array of actuators, for example by defining one or more of the following in content metadata corresponding to the image or video:• The method of resampling / resizing the image or video to an actuator array having a different aspect ratio than that of the asset;• Minimum capabilities of the actuator array required to render the image or video, including but not limited to:o The number of elements in the actuator arrayo The maximum spatial distance between elements of the actuator array; o The maximum spatial distance between actuator sub-arrays; and / or o The actuation capacity of the actuator array, such as the color gamut for light fixtures or the frequency range for haptic actuators.

[0363] A tuner may desire to control at least some aspects of the process of configuring how to Tenderer images and videos. Some tuning tools may provide a tuner with one or more of the following:• The ability to visualize the performance of the image and video in a GUI, e.g., for a range of different images, videos and resizing / scaling methods;• The ability to mark particular actuators or particular actuator arrays as capable or incapable of rendering images and videos; and / or• The ability to rank actuator arrays in order of preference when selecting an array to render an image or video.D24103W001Actuator Array Domain Rendering

[0364] In some instances, sensory content may may be created that is similar in nature to screen-anchored sensory objects, in the sense that the spatial and temporal properties of the sensory objects may be designed relative to an actuator array, such as a light strip or a light grid. In one such example, a content creator may design a sensory effect to be produced by way of a spatial sensory object traversing over a spatial domain with a particular positional trajectory. The content creator may, for example according to sensory object metadata, convey their intent for this sensory effect to be rendered on a single actuator array, for example on a single light strip. In order to create actuator array domain effects, the content creator may specify, for example according to sensory object metadata, a global position to indicate where the sensory effect is to be rendered in the playback environment.

[0365] In some examples, a control system — such as a control system that implements a Tenderer — may be configured to determine which actuator array in a playback environment will to be used to render an effect. In some such examples, the control system may be configured to map the array domain trajectory onto that actuator array according to one or more of the following:• the size of the playback environment• The position of the actuator array within the playback environment;• I’he spatial resolution of the actuator array; and / or• The positions of the actuators of the actuator array in the playback environment.

[0366] According to some examples, the control system may be configured to perform an automatic configuration process that involves normalizing the coordinates of the positions of the actuators of every actuator array in the playback environment, e.g., according to a bounding box. In some examples in which the playback environment is a room, the bounding box may have axes that are aligned with one or more axes of the room. The automatic configuration process may also involve marking actuator arrays as not capable of rendering actuator array domain content due to one or more of the following:• The spatial resolution of the array;• The position of the array.

[0367] The nomralization of positions may, in some examples, be performed as described elsewhere herein regarding the normalization of screen- anchored objects. In some examples, there may be multiple actuators mapped into an actuator array domain.D24103W001According to some such examples, these multiple actuators may be arrays themselves and may create a super- array.

[0368] A tuner may desire to control at least some aspects of the process of configuring how to Tenderer actuator array domain effects. Some tuning tools may provide a tuner with one or more of the following:• the ability to visualize the performance of actuator array domain effect rendering in a GUI.• The ability to move the effective position of an array in order to change what endpoint positions may to the array;• The ability to mark particular actuator arrays as capable or incapable of rendering array domain effects; and / or• The ability to rank actuator arrays in order of preference when selecting an array to render an actuator array domain effect.Configuring Personalization Zones

[0369] In some instances, personalization, control and configuration on a per- zone basis may be desirable, especially for playback environments in which multiple users are likely to be present. In this context, a “per-zone basis” means according to different zones, areas or volumes of a playback environment. Determining personalisation zones may, in some examples, be done automatically. In some such examples, personalisation zones may be determined in the same manner that screen domain membership is determined.Graph-Based Rendering

[0370] In some examples, sensory objects and sensory effects can be created according to graph signal processing, which may involve analyzing the underlying structure of the actuators within a playback environment in order to produce a graph in which the nodes are the controllable actuators in the playback environment. A useful product of this process is the “shift matrix” SGEWy,x Na, which may be used to propagate signals on the graph. One can define a signal, xG, on the graph comprised of the actuators in the room as xGEWA The shift matrix is a mechanism which allows a control system to propagate this signal across the graph, e.g., as follows:xG[n + 1] =GxG[n]D24103W001

[0371] In the foregoing equation, n represents the current time step and n + 1 represents the next timestep, during which the graph signal will be propagated. A graph signal may be incorporated this an actuator activation matrix, e.g., as follows:AGA =AvAB. As.

[0372] In the foregoing equation, AG= xG. The reader will appreciate that in some instances one or more elements may be missing from the foregoing equation. For example, if there are no beds being rendered there will be no AB component.

[0373] The shift matrix only contains information about the configuration of the endpoint. The shift matrix is generally not parameterized by the sensory content.

[0374] When a graph-based sensory object is rendered, rendering the first frame involves determining xG. This process may include, for example, one or more of the following processes:Setting one or more elements of xGaccording to the position of the actuators and the graph object. Elements may be set according to a proximity activation, which may in some examples be an activation caused by the position and shape of a discrete spatial object, such as a cuboid, or a sphere. In some examples, elements may be set according to a bed activation;Setting one or more elements across the entire playback environment according to a parametric function, e.g., in the manner that volumetric fields arc rendered;Setting one or more elements across the entire playback environment according to data, e.g., in the manner that volumetric textures are rendered.

[0375] In some examples, the graph-based object may set the initial state of the graph signal and the shift matrix may propagate the signal over time. According to some examples, the control system may require the graph object to control how the signal evolves on the graph overtime, e.g., as follows:xG[n + l] = WGSG%G[n]

[0376] In the foregoing equation, HGG [R / v'lX Narepresents a graph-object propagation matrix, which, when implemented, can perform one or more of the following:Control the temporal frequency of the signal being propagated on the graph;Control the spatial frequency of the signal being propagated on the graph;D24103W001Add diffusivity to the propagation mechanism;Mask or block particular paths through the graph.

[0377] A graph of actuators can be defined using an adjacency matrix WGG IR'V / lX Nathat contains weights connecting each actuator to every other actuator. In other words, WG lj represents the weight connecting the ithactuator to the jthactuator. When WG ij = WGJ ithe adjacency matrix is symmetric, and the graph is said to be undirected.

[0378] According to some examples, the control system may be configured to construct HGat render time using, for example, one or more of the following:The adjacency matrix WGand / or other matrices derived from WG, such as the (normalized) Laplacian of WG;The shift matrix SG;Object metadata;Endpoint metadata;Rendering metadata such as frame rate (so the shift matrix can be temporally controlled).

[0379] In some examples, the control system may be configured to construct WG. This process may involve analyzing the playback environment configuration according to, for example:• The positions of the actuators;• The direction in which their radiation patterns are pointed;• The direction of the axes of arrays of actuators, e.g., light strips;• The position(s) of the scrccn(s); and / or• The actual or assumed position(s) of the user(s).

[0380] According to some examples, the control system may be configured to construct an undirected graph based on the spatial proximity of the actuators only, e.g., as follows:|| Lr- r||2tf $i,j <0 otherwise

[0381] In the foregoing equations, crwrepresents a threshold that limits the number of connected nodes in the graph, L^osrepresents the position of the ithactuator and Losrepresents the position of the jthactuator. In some examples, crwmay be set adaptively soD24103W001that a minimum number of non-zero connections are made to each node. Furthermore, this may be a minimum number of connected nodes that deviate significantly in one or more dimensions. For example, < JWmay be set adaptively so that each node has at least one connection in the x, y and z dimensions. Differing significantly in one or more dimensions can be computed by taking the vector connecting the two node positions and computing the angle between that vector and each axis of the room. In some examples, alignment within 30 or 45 degrees would be sufficient. In other examples, awmay be set adaptively to limit the number of connected nodes, for example by settingwto thehighest value of WGtj and then taking the top r connections only. According to some examples, the control system may be configured to apply these methods to simply ensure that each node has just one connection in the positive and negative directions along each axis of the room. In other examples, < TWmay not be set adaptively and a threshold corresponding to a Euclidean distance in the room is set, which may be on the order of 5cm to 50 cm in some instances. In the foregoing equations, cis() represents a suitable activation function, such as:• An inverse function, e.g., us(x) = or• An exponential activation function, e.g., us(x) = exp ( I.

[0382] A “suitable” activation function may be one which enforces lower adjacency as the spatial distance increases.

[0383] Some examples may involve determining multiple graphs for the playback environment. For example, a user may wish to have a graph which is suitable for propagating signals across the playback environment along a particular axis, such as the x axis. In some such examples, the control system may be configured to construct the adjacency matrix WGX, according to the distance between the actuators in the x dimension weighted by their total distance, e.g., as follows:WGX, IJ =as2( c2( Lp°s- as(Ci ) )

[0384] In the foregoing equation, Lp°srepresents the position of the i,hactuator along the x axis, if’*0represents the position of the jthactuator along the x axis, aS2( ) represents an activation function that could differ from as(), and represents a thresholded 6t matrix. In some examples, the activations functions may be a Gaussian activation function, a triangle activation function or a rectangular activation function. c and c2arc scalars usedD24103W001to bring the arguments parsed into them onto a suitable range (e.g., from 0 to 1 for a normalized triangle activation).

[0385] In some examples, the control system may be configured to compute a thresholded distance as follows:, > 1 ^i,j if $i,j °dl, Jklnf otherwise

[0386] In the foregoing equation, adrepresents a distance threshold. In some alternative examples, the control system may be configured to only consider the x distance between actuators within a region determined by <rd, e.g., as follows:, _ I0if ^ij < °d1,7klnf otherwise

[0387] In some cases, it may be desirable to determine sub-graphs of the playback environment. For example, when the Tenderer is rendering actuator array domain effects, the control system may be configured to determine super-arrays composed of multiple actuator arrays within the playback environment by analyzing a suitably-constructed adjacency matrix. In some examples, the control system may be configured to apply one or more techniques such as spectral clustering, where the membership of actuators in each cluster can be used to construct the super-arrays.

[0388] The adjacency matrix of any graph or subgraph may, in some implementations, be further processed to produce a shift matrix SGG BW / lX Na, which may be used to propagate signals on the graph. Alternatively, the control system may be configured to construct the shift matrix independently. A simple shift graph may simply connect each node in the graph with one other node, e.g., by setting the largest element of each row of WGto be 1 and the remaining elements in the row to 0 to construct the shift matrix. This type of shift matrix is sparse.

[0389] In some cases, it may be desirable to have a diffuse shift matrix, which results in signal dispersion on the graph. In some examples, the control system may be configured to construct a diffuse shift matrix as follows:• Setting the largest k elements of the row of WGto be 1; or• Keeping the largest k elements of the row of WG; or• Keeping the largest k elements of the row of WGand normalizing the largest k elements.D24103W001

[0390] In some examples, there may be multiple instances HGand / or SGfor a range of graphs and subgraphs across the entire endpoint. According to some examples, these graphs may operate within screen- anchored or room-anchored coordinate systems.

[0391] Figure 18 is a flow diagram that outlines one example of a method that may be performed by an apparatus or system such as those disclosed herein. The blocks of method 1800, like other methods described herein, are not necessarily performed in the order indicated. In some implementation, one or more of the blocks of method 1800 may be performed concurrently. Moreover, some implementations of method 1800 may include more or fewer blocks than shown and / or described. The blocks of method 1800 may be performed by one or more devices, which may be (or may include) one or more instances of control system such as the control system 110 that is shown in Figure 1 A and described above. For example, at least some aspects of method 1800 may be performed by an instance of the control system 110 that is configured to implement the multi-sensory Tenderer of Figure 3. Some aspects of method 1800 may be performed by an instance of the control system 110 that is configured to implement the lightscape Tenderer 501 of Figure 5 or Figure 7.

[0392] In this example, method 1800 involves configuring a sensory data renderer. Here, block 1805 involves obtaining, by a control system, playback environment data corresponding to a playback environment. In this example, the playback environment data includes playback environment geometry data and actuator data for a set of controllable actuators of the playback environment. According to this example, the actuator data includes actuator position data. In some examples, the set of one or more controllable actuators may include one or more light fixtures, one or more haptic devices, one or more air flow control devices, or combinations thereof.

[0393] According to this example, block 1810 involves configuring, by the control system and based at least in part on the playback environment data, the sensory data Tenderer to render received sensory data and to produce actuator control signals for the set of controllable actuators of the playback environment.

[0394] According to some examples, configuring the sensory data Tenderer may involve playback environment scale normalization. In some examples, method 1800 may involve obtaining, by a control system, reference environment data corresponding to a reference environment, the reference environment data including reference environment geometryD24103W001data. In some such examples, configuring the sensory data renderer may be based, at least in part, on the reference environment data.

[0395] In some examples, configuring the sensory data Tenderer may involve configuring bed-actuator mapping functionality. In some such examples, the bed-actuator mapping functionality may involve mapping one or more bed channels of the received sensory data to one or more controllable actuators of the set of controllable actuators of the playback environment. According to some examples, configuring the sensory data Tenderer may involve configuring spatial masking functionality of the sensory data renderer.

[0396] In some examples, the set of controllable actuators may include a set of one or more light fixtures. In some such examples, the sensory data Tenderer may include a lightscape renderer configured to received light-based sensory data and to produce control signals for the set of one or more light fixtures.

[0397] According to some examples, configuring the sensory data renderer may involve configuring the lightscape renderer. In some examples, configuring the lightscape renderer may involve configuring extended lights rendering functionality relating to extended light volumes of one or more playback environment light fixtures of the playback environment. According to some examples, configuring the lightscape renderer may involve configuring lux density normalization functionality. In some examples, configuring the lightscapc Tenderer may involve configuring image and video object mapping functionality.

[0398] In some examples, method 1800 may involve providing, by the control system and via a user interface system, information regarding a current configuration of the sensory data renderer. According to some examples, method 1800 may involve providing, by the control system and via a user interface system, one or more user interfaces for receiving user input for altering an automatic configuration of the sensory data renderer, one or more user interfaces for manually setting one or more output values of the sensory data renderer, or combinations thereof.

[0399] Figures 19 A, 19B and 19C show examples of graphical user interfaces (GUIs) that may be provided by a tuning tool, such as a tuning software application. As with other figures provided herein, the types and numbers of elements shown in Figures 19C-19C are merely provided by way of examples. Other implementations may include more, fewer and / or different types and / or numbers of elements.

[0400] Figure 19A shows an example of a GUI 1900a. In this example, the GUI 1900a allows a user, also referred to herein as a “tuner,” to configure parameters related to screenD24103WO01domain mapping. According to this example, the GUI 1900a includes GUI areas 1901, 1903 and 1905. In this example, the GUI area 1901 shows an example of a playback environment layout or “room view,” the GUI area 1903 shows an example of a screen domain view and the GUI area 1905 shows examples of configurable settings relating to screen domain mapping with which a user can interact.

[0401] In the example shown in Figure 19 A, the GUI area 1901 shows a top-down view of the playback environment with multiple screens and users that is also shown in Figure 14. Because the elements of Figure 14 are described above, these descriptions will not be repeated here. Here, the GUI area 1901 shows the domain membership of each light and screen in the playback environment as well as actual or assumed user positions.

[0402] Here, the GUI area 1903 shows the same top-down view of a reference room that is shown in Figure 15 A. Because the elements of Figure 15 A are described above, they will not be repeated here. As noted above, the elements of Figure 15A correspond to the “main screen domain” shown in the GUI area 1901. In this example, the GUI area 1903 shows the normalized light fixture positions in the screen domain.

[0403] In this example, the GUI area 1905 includes GUI subareas 1907, 1909 and 1910. Here, the GUI subarea 1907 allows the tuner to assign light fixtures, or portions thereof, to each selected physical domain. At the time represented by Figure 19 A, the user has selected “physical domain 1” and therefore the GUI subarea 1907 allows a user to assign light fixtures, or portions thereof, to the “main screen domain” shown in the GUI area 1901. According to this example, the selection of “physical domain 1” has caused the corresponding screen domain view to appear in the GUI area 1903. In this example, the user has chosen to assign all segments of the light strip 30101 to physical domain 1.

[0404] Here, the GUI subarea 1909 allows the tuner to configure screen parameters for the selected physical domain. In this example, the GUI subarea 1909 allows the tuner to configure the x, y and z coordinates of the viewer’s position (e.g., of the viewer’s head) and of the bottom left and top right comers of the screen.

[0405] According to this example, the GUI subarea 1910 allows the tuner to select a desired type of logical to physical domain mapping, e.g., as described herein with reference to Figures 17A-17O. In this example, logical domain 1 (LD1) will map to screen domain 1 (SD1, which is physical domain 1) and logical domain 2 will map to screen domain 2.D24103W001

[0406] Figure 19B shows an example of a GUI 1900b. In this example, the GUI 1900b allows a user to configure parameters related to bed mapping. According to this example, the GUI 1900b includes GUI areas 1911 and 1925. In this example, the GUI area 1911 includes playback environment lighting representations 1913, 1915, 1917, 1919, 1921 and 1923, which arc 3D representations of the playback environment that indicate light fixtures corresponding to “up,” “down,” “left,” “right,” “front” and “rear” beds, respectively. In the playback environment lighting representations 1913-1923, light fixtures are represented as circles. Light fixtures shown as circles having a light outline correspond with the indicated bed and light fixtures represented by a circle having a dark outline do not correspond with the indicated bed.

[0407] According to this example, the GUI area 1925 shows examples of configurable settings relating to bed mapping with which a user can interact. In this example, the GUI area 1925 includes GUI subareas 1927, 1929, 1931 and 1933. According to this example, the GUI subarea 1927 includes a “method” window with which the tuner can interact in order to select an optimization method. In this example, the user has selected a least squares optimization method. According to this example, the GUI subarea 1927 includes a “bed set” window with which the tuner can interact in order to select a set of beds. In this example, the user has selected “6 bed (LRFRUD),” which means that the user has selected 6 beds in total, specifically left, right, front, rear, up and down beds. The selected beds correspond to the playback environment lighting representations shown in the GUI area 1911 and may also correspond to what is referred to elsewhere herein as a “cubic-bed” example.

[0408] In this example, the GUI subarea 1927 also includes a “cost” window with which the tuner can interact in order to select a metric with which to evaluate a bed-actuator membership. As noted above, one way to define a metric with which to evaluate a bedactuator membership that is well-suited for cubic-bed examples is based on the squared distance along the normal to the surface of each of the faces of the cubic beds. This type of method is referred to herein as a “face distance method.” In this example, the user has selected “face distance,” which means that the user has selected the face distance method that is describe above.

[0409] As noted above, some examples may involve combining cost functions in order to produce an overall cost function, e.g., as follows:L= + ABLB+D24103W001

[0410] In the foregoing equation, Lsrepresents an actuator membership cost function that will penalize ABmatrices that have all zeros in some rows, LB represents a bed membership cost function which will penalize ABmatrices that have all zeros in some columns, LM represents a bed equality cost function which will penalize ABmatrices where columns have a large variance between the sum of their rows, and As,BandMrepresent real scalars for weighting the actuator membership cost function, the bed membership cost function and the bed equality cost function, respectively. In one example, ls, 2Candmay represent Lagrange multipliers used to weight each cost function to compute the overall cost.

[0411] According to this example, the GUI subarea 1929 includes windows with which the tuner can interact in order to select weighting parameters for the least squares method that the tuner has selected in the GUI subarea 1927: here, the tuner can select As,.Band.Mby indicating values in the “actuator membership” window, the “bed membership” window and the “bed equality” window, respectively.

[0412] As noted above, some examples may involve optimizing 14 / iteratively using numerical optimization methods such as gradient descent. This iterative process may be performed by a “solver,” that is implemented by the control system. For example, the solver may be implemented via software that includes instructions to perform one or more of the optimization methods disclosed herein. Some examples may involve post-processing ABafter the iterative optimization process has converged. Some such examples involve postprocessing ABby setting the maximum value of each column to 1 and the remaining elements to zero. This will result in each actuator belonging to just one bed, which may be desirable in some instances. In some alternative examples, the “post-processing” described above may occur during the process of computing the loss functions during the iterative optimization process.

[0413] In this example, the GUI subarea 1931 includes windows with which the tuner can interact in order to select one or more light fixtures for membership to a particular bed, potentially overriding the memberships that are automatically selected by the solver.According to this example, the GUI subarea 1933 includes a window with which the tuner can interact in order to select a step size in order to tune the optimization process, e.g., so that the optimization process goes faster and / or is more stable. In general, a larger step size produces a faster optimization process and a smaller step size produces a more stable optimization process. In this example, the GUI subarea 1933 includes a virtual “start”D24103W001button for initiating the optimization process and a status window indicating the progress of the optimization process: in this example, the status window indicates that the optimization process is 100% complete.

[0414] Figure 19C shows an example of a GUI 1900c. In this example, the GUI 1900c allows a user to configure parameters related to brightness tuning. According to this example, the GUI 1900b includes GUI areas 1941 and 1945. In this example, the GUI area 1941 shows an enlarged instance of the playback environment lighting representation 1913 of Figure 19B.

[0415] According to this example, the GUI area 1945 includes GUI subareas 1947, 1949 and 1951. In this example, the GUI subarea 1947 includes a global brightness window with which a user may select an overall brightness value. In some examples, the overall brightness value may range from zero to 1.

[0416] In this example, the GUI subarea 1949 includes a light fixture selection window and a light fixture brightness window with which a user may select a light fixture and a brightness value for the light fixture. In some examples, the brightness value may range from zero to 1. In this example, the selected light fixture is a light strip. According to this example, the GUI subarea 1949 includes a window with which a user may interact in order to select whether to apply the brightness value to all segments of the selected light strip. In some such examples, if the user indicates that the brightness value should not be applied to all segments of the selected light strip, the user may be presented with an updated GUI subarea 1949 that allows the user to select brightness values for two or more portions of the light strip.

[0417] According to this example, the GUI subarea 1951 includes windows for a user to indicate Tenderer configurations. For example, the GUI subarea 1951 includes a window for the user to select whether the Tenderer should continually update output colors to the light fixtures during the tuning process. In this example, the GUI subarea 1951 also includes a window for the user to select whether light fixture calibration — also referred to herein as color management — should be applied. According to this example, the GUI subarea 1951 also includes windows for selecting red, blue and green color values. In some examples, the color values may range from zero to 1. In this example, the GUI subarea 1951 also includes a “render one frame” virtual button with which a user may initiate a Tenderer update.According to this example, the GUI subarea 1951 also includes a “connect / disconnect”D24103W001virtual button with which a user may connect and disconnect the tuning tool to the Tenderer to update output colors to the light fixtures.

[0418] The above description illustrates various embodiments of the present disclosure along with examples of how aspects of the present disclosure may be implemented. The above examples and embodiments should not be deemed to be the only embodiments, and are presented to illustrate the flexibility and advantages of the present disclosure as defined by the following claims. Based on the above disclosure and the following claims, other arrangements, embodiments, implementations and equivalents will be evident to those skilled in the art and may be employed without departing from the spirit and scope of the disclosure as defined by the claims.

[0419] Various aspects of the present disclosure may be appreciated from the following Enumerated Example Embodiments (EEEs):EEE1. A method of configuring a sensory data Tenderer, the method comprising: obtaining, by a control system, playback environment data corresponding to a playback environment, the playback environment data including playback environment geometry data and actuator data for a set of controllable actuators of the playback environment, the actuator data including actuator position data; andconfiguring, by the control system and based at least in part on the playback environment data, the sensory data Tenderer to render received sensory data and to produce actuator control signals for the set of controllable actuators of the playback environment.EEE2. The method of EEE1, further comprising obtaining, by a control system, reference environment data corresponding to a reference environment, the reference environment data including reference environment geometry data, wherein configuring the sensory data Tenderer is based, at least in part, on the reference environment data.EEE3. The method of EEE1 or EEE2, wherein configuring the sensory data Tenderer involves playback environment scale normalization.EEE4. fhe method of any one of EEE1-EEE3, wherein configuring the sensory data Tenderer involves configuring bed-actuator mapping functionality and wherein the bedactuator mapping functionality comprises mapping one or more bed channels of the received sensory data to one or more controllable actuators of the set of controllable actuators of the playback environment.D24103W001EEE5. The method of any one of EEE1-EEE4, wherein configuring the sensory data Tenderer involves configuring spatial masking functionality of the sensory data renderer. EEE6. The method of any one of EEE1-EEE5, wherein configuring the sensory data Tenderer involves configuring snapped spatial sensory object functionality of the sensory data Tenderer and wherein the snapped spatial sensory object functionality involves generating one or more actuator control signals for a snapped spatial sensory object even if no actuator control signal would otherwise have been generated based on a size and a position of the snapped spatial sensory object.EEE7. The method of any one of EEE1-EEE6, wherein configuring the sensory data Tenderer involves configuring automatic assignment of spatial sensory object functionality. EEE8. The method of any one of EEE1-EEE7, wherein configuring the sensory data Tenderer involves configuring sensory volumetric field scaling functionality.EEE9. The method of EEE8, wherein configuring the sensory data Tenderer involves configuring sensory volumetric texture functionality.EEE10. The method of EEE9, wherein configuring the sensory data Tenderer involves configuring spatial sampling of a sensory volumetric field, configuring spatial sampling of a sensory volumetric texture, or both.EEE11. The method of any one of EEE1-EEE10, wherein configuring the sensory data Tenderer involves configuring actuator array domain effects functionality and wherein the actuator array domain effects functionality involves spatial sensory object effects for a particular actuator array.EEE 12. The method of any one of EEE1-EEE11, wherein configuring the sensory data Tenderer involves configuring graph-based rendering functionality.EEE13. The method of EEE12, wherein the graph-based rendering functionality involves rendering one or more graph-based sensory objects based on a shift matrix derived from an endpoint configuration or a graph-object propagation matrix.EEE14. The method of any one of EEE1-EEE13, wherein the set controllable actuators includes one or more light fixtures, one or more haptic devices, one or more air flow control devices, or combinations thereof.D24103W001EEE15. The method of EEE14, wherein the set of controllable actuators includes a set of one or more light fixtures, wherein the sensory data Tenderer comprises a lightscape Tenderer configured to received light-based sensory data and to produce control signals for the set of one or more light fixtures, and wherein configuring the sensory data Tenderer involves configuring the lightscapc rcndcrcr.EEE16. The method of EEE15, wherein configuring the lightscape Tenderer involves configuring extended lights rendering functionality relating to extended light volumes of one or more playback environment light fixtures of the playback environment.EEE17. The method of EEE15 or EEE16, wherein configuring the lightscape Tenderer involves configuring lux density normalization functionality.EEE18. The method of any one of EEE15-EEE17, wherein configuring the lightscape Tenderer involves configuring image and video object mapping functionality.EEE19. The method of any one of EEE1-EEE18, further comprising providing, by the control system and via a user interface system, information regarding a current configuration of the sensory data renderer.EEE20. The method of any one of EEE1-EEE19, further comprising providing, by the control system and via a user interface system, one or more user interfaces for receiving user input for altering an automatic configuration of the sensory data renderer, one or more user interfaces for manually setting one or more output values of the sensory data renderer, or combinations thereof.EEE21. An apparatus configured to perform the method of any one of EEE1-EEE20. EEE22. A system configured to perform the method of any one of EEE1-EEE20. EEE23. One or more non-transitory, computer- readable media having instructions stored thereon for controlling one or more devices to perform the method of any one of EEE1-EEE20.

Claims

1. D24103W0012.CLAIMS3.What Is Claimed Is:

1. A method of configuring a sensory data tenderer, the method comprising:5.obtaining, by a control system, playback environment data corresponding to a playback environment, the playback environment data including playback environment geometry data and actuator data for a set of controllable actuators of the playback environment, the actuator data including actuator position data; and6.configuring, by the control system and based at least in part on the playback environment data, the sensory data Tenderer to render received sensory data and to produce actuator control signals for the set of controllable actuators of the playback environment.

2. The method of claim 1, further comprising obtaining, by a control system, reference environment data corresponding to a reference environment, the reference environment data including reference environment geometry data, wherein configuring the sensory data Tenderer is based, at least in part, on the reference environment data.

3. The method of claim 1 or claim 2, wherein configuring the sensory data Tenderer involves playback environment scale normalization.

4. The method of any one of claims 1-3, wherein configuring the sensory data Tenderer involves configuring bed-actuator mapping functionality and wherein the bed-actuator mapping functionality comprises mapping one or more bed channels of the received sensory data to one or more controllable actuators of the set of controllable actuators of the playback environment.

5. The method of any one of claims 1-4, wherein configuring the sensory data Tenderer involves configuring spatial masking functionality of the sensory data renderer.

6. The method of any one of claims 1-5, wherein configuring the sensory data Tenderer involves configuring snapped spatial sensory object functionality of the sensory data Tenderer and wherein the snapped spatial sensory object functionality involves generating one or more actuator control signals for a snapped spatial sensory object even if no actuator control signal would otherwise have been generated based on a size and a position of the snapped spatial sensory object.D24103W0017. The method of any one of claims 1-6, wherein configuring the sensory data Tenderer involves configuring automatic assignment of spatial sensory object functionality.

8. The method of any one of claims 1-7, wherein configuring the sensory data Tenderer involves configuring sensory volumetric field scaling functionality.

9. The method of claim 8, wherein configuring the sensory data Tenderer involves configuring sensory volumetric texture functionality.

10. The method of claim 9, wherein configuring the sensory data Tenderer involves configuring spatial sampling of a sensory volumetric field, configuring spatial sampling of a sensory volumetric texture, or both.

11. The method of any one of claims 1-10, wherein configuring the sensory data Tenderer involves configuring actuator array domain effects functionality and wherein the actuator array domain effects functionality involves spatial sensory object effects for a particular actuator array.

12. The method of any one of claims 1-11, wherein configuring the sensory data Tenderer involves configuring graph-based rendering functionality.

13. The method of claim 12, wherein the graph-based rendering functionality involves rendering one or more graph-based sensory objects based on a shift matrix derived from an endpoint configuration or a graph-object propagation matrix.

14. The method of any one of claims 1-13, wherein the set controllable actuators includes one or more light fixtures, one or more haptic devices, one or more air flow control devices, or combinations thereof.

15. The method of claim 14, wherein the set of controllable actuators includes a set of one or more light fixtures, wherein the sensory data Tenderer comprises a lightscape Tenderer configured to received light-based sensory data and to produce control signals for the set of one or more light fixtures, and wherein configuring the sensory data Tenderer involves configuring the lightscape Tenderer.

16. The method of claim 15, wherein configuring the lightscape Tenderer involves configuring extended lights rendering functionality relating to extended light volumes of one or more playback environment light fixtures of the playback environment.D24103W00117. The method of claim 15 or claim 16, wherein configuring the lightscape Tenderer involves configuring lux density normalization functionality.

18. The method of any one of claims 15-17, wherein configuring the lightscape Tenderer involves configuring image and video object mapping functionality.

19. The method of any one of claims 1-18, further comprising providing, by the control system and via a user interface system, information regarding a current configuration of the sensory data renderer.

20. The method of any one of claims 1-19, further comprising providing, by the control system and via a user interface system, one or more user interfaces for receiving user input for altering an automatic configuration of the sensory data renderer, one or more user interfaces for manually setting one or more output values of the sensory data renderer, or combinations thereof.

21. An apparatus configured to perform the method of any one of claims 1-20.

22. A system configured to perform the method of any one of claims 1-20.

23. One or more non-transitory, computer-readable media having instructions stored thereon for controlling one or more devices to perform the method of any one of claims 1-20.

Images