Color management for lightscapes

Abstract

Some examples involve obtaining light fixture data for a set of one or more controllable light fixtures, the light fixture data including light fixture color volume data for each controllable light fixture of the set of one or more controllable light fixtures. Some examples involve receiving light object data including a set of one or more light objects and corresponding light object metadata, the light object metadata including intended light object color data and light object spatial data. Some examples involve rendering the light object data to produce one or more light fixture control signals, wherein the rendering is based at least in part on the light fixture color volume data and the light object metadata. Some examples involve providing the one or more light fixture control signals to one or more controllable light fixtures of the set of controllable light fixtures.

Description

D24099W001COLOR MANAGEMENT FOR LIGHTSCAPESCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of priority from U.S. Provisional Application No.63 / 720,644, filed on November 14, 2024, and U.S. Provisional Application No. 63 / 784,894 filed on April 7, 2025, each of which is incorporated by reference herein in its entirety.TECHNICAL FIELD

[0002] The present disclosure relates to providing multi-sensory (MS) experiences, which may also be referred to herein as multi-modal (MM) experiences, some of which include light-based sensory experiences.BACKGROUND

[0003] Unless otherwise indicated herein, the approaches described in this section are not prior art to the claims in this application and are not admitted as prior art by inclusion in this section.

[0004] Media content delivery has generally focused on audio and video experiences. There has been limited delivery of multi-sensory content due to the bespoke nature of actuation. For example, luminaires, which are also referred to herein as light fixtures or simply as lights, are used extensively as an expression of art and function for concerts. However, each installation is designed specifically for a unique set of luminaires. Delivering a lighting design beyond the set of light fixtures the system was designed for is generally not feasible. Other systems that attempt to deliver light experiences, also referred to herein as “lightscapes,” simply do so by extending the screen visuals algorithmically, but are not specifically authored.SUMMARY

[0005] At least some aspects of the present disclosure may be implemented via methods. In some instances, the methods may be implemented, at least in part, by a control system such as those disclosed herein. Some disclosed methods involve controlling a set of one or more controllable light fixtures of a playback environment.D24099W001

[0006] Some disclosed methods may involve obtaining, by a control system, light fixture data for the set of one or more controllable light fixtures. In some examples, the light fixture data may include light fixture color volume data for each controllable light fixture of the set of one or more controllable light fixtures.

[0007] Some disclosed methods may involve receiving, by the control system, light object data including a set of one or more light objects and corresponding light object metadata. According to some examples, the light object metadata may include intended light object color data and light object spatial data.

[0008] Some disclosed methods may involve rendering, by the control system, the light object data to produce one or more light fixture control signals. In some examples, the rendering may be based at least in part on the light fixture color volume data and the light object metadata.

[0009] Some disclosed methods may involve providing, by the control system, the one or more light fixture control signals to one or more controllable light fixtures of the set of controllable light fixtures.

[0010] In some examples, a first light fixture color volume for a first controllable light fixture of the set of one or more controllable light fixtures may differ from a second light fixture color volume for a second controllable light fixture of the set of one or more controllable light fixtures. In some such examples, the rendering may involve determining an overlapping light fixture color volume of the first light fixture color volume and the second light fixture color volume and providing the one or more light fixture control signals according to the overlapping light fixture color volume.

[0011] According to some examples, the rendering may be based at least in part on a viewer position, a viewer orientation, a distance between the first controllable light fixture and the second controllable light fixture, a distance between the first controllable light fixture and a display screen, a distance between the second controllable light fixture and the display screen, or combinations thereof.

[0012] In some examples, the rendering may be based at least in part on minimizing a difference between an intended light object color and a color produced by a controllable light fixture. According to some examples, the rendering may be based at least in part on minimizing a difference between a first color produced by a first controllable light fixture and a second color produced by a second controllable light fixture. In some examples, the rendering may be based at least in part on a just-noticeable-difference model.D24099W001

[0013] Some disclosed methods may involve obtaining, by the control system, display capability data for at least one display in the playback environment. According to some examples, the display capability data may include display color volume data. In some such examples, the rendering may be based at least in part on the display color volume data. According to some examples, the rendering may be based at least in part on minimizing a difference between a color produced by the display and a color produced by a controllable light fixture.

[0014] Some disclosed methods may involve determining, by the control system, an overlapping light fixture / display color volume between at least a first light fixture color volume and a display color volume and determining the one or more light fixture control signals according to the overlapping light fixture / display color volume. Some examples may involve determining, by the control system, an overlapping light fixture / display color volume between at least a first light fixture color volume and a display color volume and controlling, by the control system, the display according to the overlapping light fixture / display color volume.

[0015] Some disclosed methods may involve determining, by the control system, a modified white point based, at least in part, on minimizing differences in colors provided by each controllable light fixture of the set of one or more controllable light fixtures. In some such examples, the rendering may involve applying the modified white point. In some examples, determining the modified white point may be based, at least in part, on controlling one or more red / blue / green (RGB) outputs for at least one controllable light fixture to reduce color variation across the set of one or more controllable light fixtures.

[0016] Some disclosed methods may involve determining, by the control system, a color volume mapping procedure. In some such examples, the rendering may be based, at least in part, on the color volume mapping procedure. According to some examples, determining the color volume mapping procedure may involve minimizing a difference between an intended light object color coordinate in a color space and a light fixture color coordinate in the color space.

[0017] Some or all of the operations, functions and / or methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more computer-readable non-transitory media. Such non-transitory media may include one or more memory devices such as those described herein, including but not limited to one or more random access memory (RAM) devices, read-only memory (ROM) devices, etc.Accordingly, some innovative aspects of the subject matter described in this disclosure canD24099W001be implemented in one or more computer-readable non-transitory media having software stored thereon.

[0018] At least some aspects of the present disclosure may be implemented via apparatus. For example, one or more devices may be capable of performing, at least in part, the methods disclosed herein. In some implementations, an apparatus may include an interface system and a control system. The control system may include one or more general purpose single- or multi-chip processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic, discrete hardware components, or combinations thereof. The control system may be configured to perform some or all of the disclosed methods.

[0019] Details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages will become apparent from the description, the drawings, and the claims. Note that the relative dimensions of the following figures may not be drawn to scale.BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Disclosed embodiments now be described, by way of example only, with reference to the accompanying drawings.

[0021] Figure 1 is a block diagram that shows examples of components of an apparatus capable of implementing various aspects of this disclosure.

[0022] Figure 2 shows example elements of an endpoint.

[0023] Figure 3 shows examples of actuator elements.

[0024] Figure 4 shows example elements of a system for the creation and playback of multi-sensory (MS) experiences.

[0025] Figure 5 shows example elements of another system for the creation and playback of MS experiences.

[0026] Figure 6 shows an example of a graphical user interface (GUI) that may be presented by a display device of the lightscape creation tool of Figure 5.

[0027] Figure 7 shows another example of a GUI that may be presented by a display device of the lightscape creation tool of Figure 5.D24099W001

[0028] Figure 8A shows an example of color volumes of two light fixtures of a playback environment.

[0029] Figure 8B shows example elements of a system configured for receiving multi-sensory (MS) content and preparing the MS content for rendering in a playback environment.

[0030] Figure 9 is a flow diagram that outlines one example of a method that may be performed by an apparatus or system such as those disclosed herein.DETAILED DESCRIPTION

[0031] Described herein are techniques related to providing multi-sensory media content. In the following description, for purposes of explanation, numerous examples and specific details are set forth in order to provide a thorough understanding of the present disclosure. It will be evident, however, to one skilled in the art that the present disclosure as defined by the claims may include some or all of the features in these examples alone or in combination with other features described below, and may further include modifications and equivalents of the features and concepts described herein.

[0032] In the following description, various methods, processes and procedures are detailed. Although particular steps may be described in a certain order, such order is mainly for convenience and clarity. A particular step may be repeated more than once, may occur before or after other steps (even if those steps are otherwise described in another order), and may occur in parallel with other steps. A second step is required to follow a first step only when the first step must be completed before the second step is begun. Such a situation will be specifically pointed out when not clear from the context.

[0033] In this document, the terms “and”, “or” and “and / or” are used. Such terms are to be read as having an inclusive meaning. For example, “A and B” may mean at least the following: “both A and B”, “at least both A and B”. As another example, “A or B” may mean at least the following: “at least A”, “at least B”, “both A and B”, “at least both A and B”. As another example, “A and / or B” may mean at least the following: “A and B”, “A or B”. When an exclusive-or is intended, such will be specifically noted (e.g., “either A or B”, “at most one of A and B”).

[0034] This document describes various processing functions that are associated with structures such as blocks, elements, components, circuits, etc. In general, these structuresD24099W001may be implemented by one or more processors controlled by one or more computer programs.

[0035] As noted above, media content delivery has generally been focused on audio and video experiences. There has been limited delivery of multi-sensory (MS) content due to the customized nature of actuation.

[0036] This application describes methods for extending the creative palette for content creators, allowing spatial, MS experiences to be created and delivered at scale. Some such methods involve the introduction of new layers of abstraction, in order to allow authored MS experiences to be delivered to different endpoints, with different types of fixtures or actuators. As used herein, the term “endpoint” is synonymous with “playback environment” or simply “environment,” meaning an environment that includes one or more actuators that may be used to provide an MS experience. Such endpoints may include a room, such as the living room of a home, a car, a cinema, a night club or other venue, etc. Some disclosed methods involve the creation, delivery and / or rendering of object-based sensory data, which may include sensory objects and corresponding sensory metadata. This abstraction allows creative intent to be implemented in an object-based format that does not require prior knowledge of the specific controller actuation, thereby enabling greater flexibility and scalability of fixtures and actuators across endpoints. An MS experience provided via objectbased sensory data may be referred to herein as a “flexibly-scaled MS experience.”

[0037] Figure 1 is a block diagram that shows examples of components of an apparatus capable of implementing various aspects of this disclosure. As with other figures provided herein, the types and numbers of elements shown in Figure 1 are merely provided by way of example. Other implementations may include more, fewer and / or different types and numbers of elements. According to some examples, the apparatus 101 may be, or may include, a device that is configured for performing at least some of the methods disclosed herein, such as a smart audio device, a laptop computer, a cellular telephone, a tablet device, a smart home hub, etc. In some such implementations the apparatus 101 may be, or may include, a server that is configured for performing at least some of the methods disclosed herein.

[0038] In this example, the apparatus 101 includes at least an interface system 105 and a control system 110. In some implementations, the control system 110 may be configured for performing, at least in part, the methods disclosed herein. The control system 110 may, inD24099W001some implementations, be configured for receiving, via the interface system 105, receiving, by a control system, a content bitstream including encoded object-based sensory metadata, The encoded object-based sensory metadata may correspond to sensory effects such as lighting, haptics, airflow, one or more positional actuators, or combinations thereof, to be provided by a plurality of sensory actuators in an environment. In some implementations, the control system 110 may be configured for extracting object-based sensory metadata from the content bitstream and for providing the object-based sensory metadata to a sensory renderer.

[0039] According to some examples, the object-based sensory metadata may include sensory spatial metadata indicating at least a spatial position for rendering the object-based sensory metadata within the environment, an area for rendering the object-based sensory metadata within the environment, or combinations thereof. In some implementations, the object-based sensory metadata does not correspond to any particular sensory actuator in the environment. In some examples, the object-based sensory metadata may include abstracted sensory reproduction information allowing the sensory renderer to reproduce authored sensory effects, which also may be referred to herein as intended sensory effects, via various sensory actuator types, via various numbers of sensory actuators and from various sensory actuator positions in the environment.

[0040] In some examples, the content bitstream also may include encoded audio objects synchronized with the encoded object-based sensory metadata. The audio objects may include audio signals and corresponding audio object metadata. In some such implementations, the control system 110 may be configured for extracting audio objects from the content bitstream and for providing the audio objects to an audio renderer. According to some examples, the audio objects may include audio signals and corresponding audio object metadata. The audio object metadata may include at least audio object spatial metadata indicating an audio object spatial position for rendering the audio signals within the environment.

[0041] The interface system 105 may include one or more network interfaces and / or one or more external device interfaces (such as one or more universal serial bus (USB) interfaces). According to some implementations, the interface system 105 may include one or more wireless interfaces. The interface system 105 may include one or more devices for implementing a user interface, such as one or more microphones, one or more speakers, a display system, a touch sensor system and / or a gesture sensor system. In some examples, theD24099W001interface system 105 may include one or more interfaces between the control system 110 and a memory system, such as the optional memory system 115 shown in Figure 1. However, the control system 110 may include a memory system in some instances.

[0042] The control system 110 may, for example, include a general purpose single- or multichip processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, and / or discrete hardware components.

[0043] In some implementations, the control system 110 may reside in more than one device. For example, a portion of the control system 110 may reside in a device within an environment (such as a laptop computer, a tablet computer, a smart audio device, etc.) and another portion of the control system 110 may reside in a device that is outside the environment, such as a server. In other examples, a portion of the control system 110 may reside in a device within an environment and another portion of the control system 110 may reside in one or more other devices of the environment.

[0044] Some or all of the methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include memory devices such as those described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, etc. The one or more non-transitory media may, for example, reside in the optional memory system 115 shown in Figure 1 and / or in the control system 110.Accordingly, various innovative aspects of the subject matter described in this disclosure can be implemented in one or more non-transitory media having software stored thereon. The software may, for example, include instructions for controlling at least one device to process audio data. The software may, for example, be executable by one or more components of a control system such as the control system 110 of Figure 1.

[0045] In some examples, the apparatus 101 may include the optional microphone system 120 shown in Figure 1. The optional microphone system 120 may include one or more microphones. In some implementations, one or more of the microphones may be part of, or associated with, another device, such as a speaker of the speaker system, a smart audio device, etc.D24099W001

[0046] According to some implementations, the apparatus 101 may include the optional actuator system 125 shown in Figure 1. The optional actuator system 125 may include one or more loudspeakers, one or more haptic devices, one or more light fixtures, also referred to herein as luminaires, one or more fans or other air-moving devices, one or more display devices, including but not limited to one or more televisions, one or more positional actuators, one or more other types of devices for providing an MS experience, or combinations thereof. The term “light fixture” as used herein refers generally to various types of light sources, including individual light sources, groups of light sources, light strips, etc. A “light fixture” may be moveable, and therefore the word “fixture” in this context does not mean that a light fixture is necessarily in a fixed position in space. The term “positional actuators” as used herein refers generally to devices that are configured to change a position or orientation of a person or object, such as motion simulator seats. Loudspeakers may sometimes be referred to herein as “speakers.” In some implementations, the optional actuator system 125 may include a display system including one or more displays, such as one or more light-emitting diode (LED) displays, one or more organic light-emitting diode (OLED) displays, etc. In some examples wherein the apparatus 101 includes a display system, the optional sensor system 130 may include a touch sensor system and / or a gesture sensor system proximate one or more displays of the display system. According to some such implementations, the control system 110 may be configured for controlling the display system to present a graphical user interface (GUI), such as a GUI related to implementing one of the methods disclosed herein.

[0047] In some implementations, the apparatus 101 may include the optional sensor system 130 shown in Figure 1. The optional sensor system 130 may include a touch sensor system, a gesture sensor system, one or more cameras, etc.

[0048] This application describes methods for creating and delivering a flexibly scaled multi-sensory (MS) immersive experience (MSIE) to different playback environments, which also may be referred to herein as endpoints. Such endpoints may include a room, such as the living room of a home, a car, a cinema, a night club or other venue, an AR / VR headset, a PC, a mobile device, etc.

[0049] Figure 2 shows example elements of an endpoint. In this example, the endpoint is a living room 1001 containing multiple actuators 008, some furniture 1010 and a person 1000 — also referred to herein as a user — who will consume a flexibly-scaled MS experience.D24099W001Actuators 008 are devices capable of altering the environment 1001 that the user 1000 is in. Actuators 008 may include one or more televisions or other display devices, one or more luminaires — also referred to herein as light fixtures — one or more loudspeakers, etc.

[0050] The number of actuators 008, the arrangement of actuators 008 and the capabilities of actuators 008 in the space 1001 may vary significantly between different endpoint types. For example, the number, arrangement and capabilities of actuators 008 in a car will generally be different from the number, arrangement and capabilities of actuators 008 in a living room, a night club, etc. In many implementations, the number, arrangement and / or capabilities of actuators 008 may vary significantly between different instances of the same type, e.g., between a small living room with 2 actuators 008 and a large living room with 16 actuators 008. The present disclosure describes various method for creating and delivering flexibly-scaled MSIEs to these non-homogenous endpoints.

[0051] Figure 3 shows examples of actuator elements. In this example, the actuator is a luminaire 1100, which includes a network module 1101, a control module 1102 and a light emitter 1103. According to this example, the light emitter 1103 includes one or more lightemitting devices, such as light-emitting diodes, which are configured to emit light into an environment in which the luminaire 1100 resides. In this example, the network module 1101 is configured to provide network connectivity to one or more other devices in the space, such as a device that sends commands to control the emission of light by the luminaire 1100. According to this example, the control module 1102 is configured to receive signals via the network module 1101 and to control the light emitter 1103 accordingly.

[0052] Other examples of actuators also may include a network module 1101 and a control module 1102, but may include other types of actuating elements. Some such actuators may include one or more loudspeakers, one or more haptic devices, one or more fans or other airmoving devices, one or more positional actuators, one or more display devices, etc.

[0053] Figure 4 shows example elements of a system for the creation and playback of multi-sensory (MS) experiences. As with other figures provided herein, the types and numbers of elements shown in Figure 4 are merely provided by way of example. Other implementations may include more, fewer and / or different types and numbers of elements. According to some examples, system 300 may be, or may include, one or more devices configured for performing at least some of the methods disclosed herein. In some examples, system 300D24099W001may include one or more instances of the control system 110 of Figure 1 that are configured for performing at least some of the methods disclosed herein.

[0054] According to the examples in the present disclosure, creating and providing an objectbased MS Immersive Experience (MSIE) approach involves the application of a suite of technologies for creation, delivery and rendering of object-based sensory data, which may include sensory objects and corresponding sensory metadata, to the actuators 008. Some examples are described in the following paragraphs.

[0055] Object-Based Representation: In various disclosed implementations, multi-sensory (MS) effects are represented using what may be referred to herein as “sensory objects.” According to some such implementations, properties such as layer-type and priority may be assigned to and associated with attached to each sensory object, enabling content creators’ intent to be represented in the rendered experiences. Detailed examples of sensory object properties are described below.

[0056] In this example, system 300 includes a content creation tool 000 that is configured for designing multi-sensory (MS) immersive content and for outputting object-based sensory data 005, either separately or in conjunction with corresponding audio data 011 and / or video data 012, depending on the particular implementation. The object-based sensory data 005 may include time stamp information, as well as information indicating the type of sensory object, the sensory object properties, etc. In this example, the object-based sensory data 005 is not “channel-based” data that corresponds to one or more particular sensory actuators in a playback environment, but instead is generalized for a wide range of playback environments with a wide range of actuator types, numbers of actuators, etc. In some examples, the objectbased sensory data 005 may include object-based light data, object-based haptic data, objectbased air flow data, or object-based positional actuator data, object-based olfactory data, object-based smoke data, object based data for one or more other types of sensor effects, or combinations thereof. According to some examples, the object-based sensory data 005 may include sensory objects and corresponding sensory metadata. For example, if the objectbased sensory data 005 includes object-based light data, the object-based light data may include light object position metadata, light object color metadata, light object size metadata, light object intensity metadata, light object shape metadata, light object diffusion metadata, light object gradient metadata, light object priority metadata, light object layer metadata, or combinations thereof. Although the content creation tool 000 is shown providing a stream ofD24099W001object-based sensory data 005 to the experience player 002 in this example, in alternative examples the content creation tool 000 may produce object-based sensory data 005 that is stored for subsequent use. Examples of graphical user interfaces for a light-object-based content creation tool are described below.

[0057] MS Object Renderer: Various disclosed implementations provide a renderer that is configured render MS effects to actuators in a playback environment. According to this example, system 300 includes an MS renderer 001 that is configured to render object-based sensory data 005 to actuator control signals 310, based at least in part on environment and actuator data 004. In this example, the MS renderer 001 is configured to output the actuator control signals 310 to MS controllers 003, which are configured to control the actuators 008. In some examples, the MS renderer 001 may be configured to receive light objects and object-based lighting metadata indicating an intended lighting environment, as well as lighting information regarding a local lighting environment. The lighting information is one general type of environment and actuator data 004, and may include one or more characteristics of one or more controllable light sources in the local lighting environment. In some examples, the MS renderer 001 may be configured to determine a drive level for each of the one or more controllable light sources that approximates the intended lighting environment. Some alternative examples may include a separate renderer for each type of actuator 008, such as one renderer for light fixtures, another renderer for haptic devices, another renderer for air flow devices, etc. According to some examples, the MS renderer 001 (or one of the MS controllers 003) may be configured to output the drive level to at least one of the controllable light sources. In some implementations, the MS renderer 001 may be configured to adapt to changing conditions. Some examples of MS renderer 001 implementations are described in more detail below.

[0058] The environment and actuator data 004 may include what are referred to herein as “room descriptors” that describe actuator locations (e.g., according to an x,y,z coordinate system or a spherical coordinate system). In some examples, the environment and actuator data 004 may indicate actuator orientation and / or placement properties (e.g., directional and north-facing, omnidirectional, occlusion information, etc.). According to some examples, the environment and actuator data 004 may indicate actuator orientation and / or placement properties according to a 3x3 matrix, in which three elements (for example, the elements of the first row) represent spatial position (x,y,z), three other elements (for example, the elements of the second row) represent orientation (roll, pitch, yaw), and three other elementsD24099W001(for example, the elements of the third row) indicate a scale or size (sx, sy, sz). In some examples, the environment and actuator data 004 may include device descriptors that describe the actuator properties relevant to the MS Renderer 001, such as intensity range and color gamut of a light fixture, the air flow speed range and direction(s) for an air-moving device, etc.

[0059] In this example, system 300 includes an experience player 002 that is configured to receive object-based sensory data 005’, audio data 011’ and video data 012’, to provide the object-based sensory data 005 to the MS Renderer 001, to provide the audio data 011 to the audio Renderer 006 and to provide the video data 012 to the video Renderer 007. In this example, the reference numbers for the object-based sensory data 005’, audio data 011’ and video data 012’ received by the experience player 002 include primes (‘). in order to suggest that the data may in some instances be encoded. Likewise, the object-based sensory data 005, audio data 011 and video data 012 output by the experience player 002 do not include primes, in order to suggest that the data may in some instances have been decoded by the experience player 002. According to some examples, the experience player 002 may be a media player, a game engine or personal computer or mobile device, or a component integrated in an television, DVD player, sound bar, set top box, or a service provider media device such as a Chromecast, Apple TV device, or Amazon Fire TV. In some examples, the experience player 002 may be configured to receive encoded object-based sensory data 005 along with encoded audio data 011 and / or encoded video data 012. In some such examples, the encoded object-based sensory data 005’ may be received as part of the same bitstream with the encoded audio data 011’ and / or the encoded video data 012’. Some examples are described in more detail below. According to some examples, the experience player 002 may be configured to extract the object-based sensory data 005’ from the content bitstream and to provide decoded object-based sensory data 005 to the MS Renderer 001, to provide decoded audio data 011 to the audio Renderer 006 and to provide decoded video data 012 to the video Renderer 007. In some examples, time stamp information in the object-based sensory data 005’ may be used — for example, by the experience player 102, the MS Renderer 001, the audio Renderer 106, the video Renderer 107, or all of them — to synchronize effects relating to the object-based sensory data 005’ with the audio data 111’ and / or the video data 112’, which may also include time stamp information.

[0060] According to this example, system 300 includes MS controllers 003 that are configured to communicate with a variety of actuator types using application programD24099W001interfaces (APIs) or one or more similar interfaces. Generally speaking, each actuator will require a specific type of control signal to produce the desired output from the Tenderer. According to this example, the MS controllers 003 are configured to map outputs from the MS Renderer 001 to control signals for each actuator. For example, a Philips Hue™ light bulb receives control information in a particular format to turn the light on, with a particular saturation, brightness and hue, and a digital representation of the desired drive level.

[0061] In some examples, room descriptors also may describe the size and orientation of the playback environment itself, to establish a relative or absolute coordinate system to which all objects are positioned. For example, in a living room a display screen may be regarded as the front, in some instances the front and center, and the floor and ceiling may be regarded as the vertical bounds. In some such examples, the room descriptors also may also indicate bounds corresponding with the left, right, front, and rear, walls relative to the front position.According to some examples, the room descriptor also may be provided in terms of a matrix, such as a 3x3 matrix. This room descriptor information is useful in describing the physical dimensions of the playback environment, for example in physical units of distance such as meters. In some such examples, sensory object locations, sensory object sizes, and sensory object orientations may be described in units that are relative to the room size, for example in a range from - 1 to 1. Room descriptors may also describe a preferred viewing position, in some instances according to a matrix.

[0062] The types, numbers and arrangements of the actuators 008 will generally vary according to the particular implementation. In some examples, actuators 008 may include lights and / or light strips (also referred to herein as “luminaires”), vibrational motors, air flow generators, positional actuators, or combinations thereof.

[0063] Similarly, the types, numbers and arrangements of the loudspeakers 009 and the display devices 010 will generally vary according to the particular implementation. In the examples shown in Figure 4, audio data Oil and video data 012 are rendered by the audio Renderer 006 and the video Renderer 007 to the loudspeakers 009 and display devices 010, respectively.

[0064] As noted above, according to some implementations the system 300 may include one or more instances of the control system 110 of Figure 1 that are configured for performing at least some of the methods disclosed herein. In some such examples, one instance of the control system 110 may implement the content creation tool 000 and another instance of theD24099W001control system 110 may implement the experience player 002. In some examples, one instance of the control system 110 may implement the audio Renderer 006, the video Renderer 007, the multi-sensory renderer 001, or combinations thereof. According to some examples, an instance of the control system 110 that is configured to implement the experience player 002 may also be configured to implement the audio Renderer 006, the video Renderer 007, the multi-sensory renderer 001, or combinations thereof.D24099W001Multi-Sensory Rendering Synchronization

[0065] Object-based MS rendering involves different modalities being rendered flexibly to the endpoint / playback environment. Endpoints have differing capabilities according to various factors, including but not limited to the following:• The number of actuators,• The modalities of those actuators (e.g., light fixture vs. air flow control device vs. haptic device);• The types of those actuators (e.g., a white smart light vs. a red / green / blue (RGB) or red / green / blue / white (RGBW) or a red / green / blue / cool white / warm white (RGBcWwW) smart light, or a haptic vest vs. a haptic seat cushion) and• The location / layout of those actuators.

[0066] In order to render object-based sensory content to any endpoint, some processing of the object signals, e.g. intensities, colors, patterns etc., will generally need to be done. The processing of each modality’s signal path should not alter the relative phase of certain features within the object signals. For example, suppose that a lightning strike is presented in both the haptics and lightscape modalities. The signal processing chain for the corresponding actuator control signals should not result in a time delay of either type of sensory object signal — haptic or light — sufficient to alter the perceived synchronization of the two modalities. The level of required synchronization may depend on various factors, such as whether the experience is interactive and what other modalities are involved in the experience. Maximum time difference values may, for example, range from approximately 10ms to 100ms, depending on the particular context.Combinations of Lights, Airflow and HapticsVehicle Examples

[0067] The following examples are described with reference to a car, but are also applicable to other vehicles, such as trucks, vans, etc. In some examples, there may be a user interface on the steering wheel or on a touchscreen near or in the dashboard. According to some examples, the following actuators may be present in the car:1. Individually addressable lights, spatially distributed around the car as follows: o on the dashboard;o under the footwells;D24099W001o on the doors; ando in the center console.2. Individually controllable air conditioning / heating outlets distributed around the car as follows:o In the front dashboard;o Under the footwells;o In the center console facing the rear seats;o On the side pillars;o In the seats; ando Directed to the windscreens (for defogging).3. Individually controllable seats with vibro-tractile haptics; and4. Individually controllable floor mats with vibro-tactile haptics.

[0068] In this example, the modalities supported by these actuators include the following:• Lights across the individually addressable LEDs in the car, plus the indicator lights on the dash and steering wheel;• Air flow via the controllable air conditioning vents;• Haptics, including:o Steering wheel: tactile vibration feedback;o Dash touchscreen: tactile vibration feedback and texture rendering; and o Seats: tactile vibrations and movement.

[0069] In one example, a live music stream is being rendered to four users sitting in the front seats. In this example, the MS Renderer 001 attempts to optimize the experience for multiple viewing positions. During the build-up before the artist has taken the stage and the previous acts have finished, the content contains:• Interlude music;• Low intensity lighting; and• Haptic content representing the moshing of the crowd.

[0070] In addition to the rendered audio and video stream, the light content contains ambient light objects that are moving slowly around the scene. These may be rendered using one of the ambient layer methods disclosed herein, for example such that there is no spatial priority given to any user’s perspective. In some examples, the haptic content may be spatially concentrated in the lower time-frequency spectrum and may be rendered only by the vibro-tactile motors in the floor mats.D24099W001

[0071] According to this example, pyrotechnic events during the music stream correspond to multi-sensory-sensory content including:• Light objects that spatially correspond to the location of the pyrotechnics at the event; and• Haptic objects to reinforce the dynamism of the pyrotechnics via a shockwave effect.

[0072] In this example, the MS Renderer 001 renders both the light objects and the haptic objects spatially. Light objects may, for example, be rendered in the car such that each person in the car perceives the light objects to come from the left if the pyrotechnics content is located at the left of the scene. In this example, only lights on the left of the car are actuated. Haptics may be rendered across both the seats and floor mats in a way that conveys directionality to each user individually.

[0073] At the end of the concert the pyrotechnics are present in the audio content and both pyrotechnics and confetti are present in the video content. In addition to rendering light objects and haptic objects corresponding to the pyrotechnics as above, the effect of the confetti firing may be rendered using the airflow modality. For example, the individually controllable air flow vents of the HVAC system may be pulsed.Living Room Examples

[0074] In this implementation, in addition to an audio / visual (AV) system that includes multiple loudspeakers and a television, the following actuators and related controls are available in the living room:• A haptics vest that the user — also referred to as a player — is wearing;• Haptics shakers mounted to the seat in which the player is sitting;• A (haptics) controllable smart watch;• Smart lights spatially distributed around the room;• A wireless controller; and• An addressable air-flow bar (AFB), which includes an array of individually controllable fans directed to the user (similar to HVAC vents in the front dashboard of a car).

[0075] In this example, the user is playing a first person shooter game and the game contains a scene in which a destructive hurricane moves through the level. As it does so, in-gameD24099W001objects are thrown around and some hit the player. Haptics objects rendered by the MS Renderer 001 cause a shockwave effect to be provided through all of the haptics devices that the user can perceive. The actuator control signals sent to each device may be optimized according to the intensity of the impact of the in-game objects, the direction(s) of the impact and the capabilities and location of each actuator (as described earlier).

[0076] At a time before the user is struck by an in-game object, the multi-sensory content contains a haptic object corresponding to a non-spatial rumble, one or more airflow objects corresponding to directional airflow; and one or more light objects corresponding to lightning. The MS Renderer 001 renders the non-spatial rumble to the haptics devices. The actuator control signals sent to each haptics device may be rendered such that the ensemble of actuator control signals across the haptics array is congruent in perceived onset time, intensity and frequency. In some examples, the frequency content of the actuator control signals sent to the smart watch may be low -pass filtered, so that they are congruent with the frequency-limited capability of the vest, which is proximate to the watch. The MS Renderer 001 may render the one or more airflow objects to actuator control signals for the AFB such that the air flow in the room is congruent with the location and look direction of the player in the game, as well as the hurricane direction itself. Lightning may be rendered across all modalities as (1) a white flash across lights that are located in suitable locations, e.g., in or on the ceiling; and (2) an impulsive rumble in the user’s wearable haptics and seat shaker.

[0077] When the user is struck by an in-game object, a directional shockwave may be rendered to the haptics devices. In some examples, a corresponding airflow impulse may be rendered. According to some examples, a damage take effect, indicating the amount of damage caused to the player by being struck by the in-game object, may be rendered by the lights.

[0078] In some such examples, signals may be rendered spatially to the haptics devices such that a perceived shockwave moves across the player’s body and the room. The MS Renderer 001 may provide such effects according to actuator location information indicating the haptics devices locations relative to one another. The MS Renderer 001 may provide the shockwave vector and position according to the actuator location information in addition to actuator capability information. According to some examples, a non-directional air flow impulse may be rendered, e.g., all the air vents of the AFB may be turned up briefly to reinforce the haptic modality. In some examples, at the same time, a red vignette may beD24099W001rendered to the light strip surrounding the TV, indicating to the player that the player took damage in the game.

[0079] Figure 5 shows example elements of another system for the creation and playback of MS experiences. As with other figures provided herein, the types and numbers of elements shown in Figure 5 are merely provided by way of example. Other implementations may include more, fewer and / or different types and numbers of elements. According to some examples, system 500 may be, or may include, one or more devices configured for performing at least some of the methods disclosed herein. In some examples, system 500 may include one or more instances of the control system 110 of Figure 1 that are configured for performing at least some of the methods disclosed herein.

[0080] According to this example, the system shown in Figure 5 is an instance of the system shown in Figure 4. In this example, the system shown in Figure 5 is a “lightscape” embodiment in which vision (video), audio and light effects are combined to create the MS experience.

[0081] In this example, system 500 includes a lightscape creation tool 100, which is an instance of the content creation tool 000 that is described with reference to Figure 4. The lightscape creation tool 100 is configured for designing and outputting object-based light data 505’, either separately or in conjunction with corresponding audio data 111’ and / or video data 112’, depending on the particular implementation. The object-based light data 505’ may include time stamp information, as well as information indicating light object properties, etc. In some instances, the time stamp information may be used to synchronize effects relating to the object-based light data 505’ with the audio data 111’ and / or the video data 112’, which also may include time stamp information.

[0082] In this example, the object-based light data 505’ includes light objects and corresponding light metadata. For example, the object-based light data may include light object position metadata, light object color metadata, light object size metadata, light object intensity metadata, light object shape metadata, light object diffusion metadata, light object gradient metadata, light object priority metadata, light object layer metadata, or combinations thereof. Although the content creation tool 100 is shown providing a stream of object-based light data 505’ to the experience player 102 in this example, in alternative examples the content creation tool 100 may produce object-based light data 505’ that is stored forsubsequent use. Examples of graphical user interfaces for a light-object-based content creation tool are described below.

[0083] In this example, system 500 includes an experience player 102 that is configured to receive object-based light data 505’, audio data 111’ and video data 112’, to provide the object-based light data 505 to the lightscape renderer 501, to provide the audio data 111 to the audio renderer 106 and to provide the video data 112 to the video renderer 107.According to some examples, the experience player 102 may be a media player, a game engine or personal computer or mobile device, or a component integrated in an television, DVD player, sound bar, set top box, or a service provider media device such as a Chromecast, Apple TV device, or Amazon Fire TV. In some examples, the experience player 002 may be configured to receive encoded object-based light data 505’ along with encoded audio data 111’ and / or encoded video data 112’. In some such examples, the encoded object-based light data 505’ may be received as part of the same bitstream with the encoded audio data 111’ and / or the encoded video data 112’. Some examples are described in more detail below. According to some examples, the experience player 102 may be configured to extract the object-based light data 505 from the content bitstream and to provide decoded object-based light data 505 to the lightscape renderer 501, to provide decoded audio data 111 to the audio renderer 106 and to provide decoded video data 112 to the video renderer 107. In some examples, the experience player 002 may be configured to allow control of configurable parameters in the lightscape renderer 501, such as immersion intensity. Some examples are described below.

[0084] In some examples, room descriptors of the environment and light fixture data 104 may describe the size and orientation of the playback environment itself, to establish a relative or absolute coordinate system to which all objects are positioned. For example, in a living room a display screen may be regarded as the front, in some instances the front and center, and the floor and ceiling may be regarded as the vertical bounds. In some such examples, the room descriptors also may also indicate bounds corresponding with the left, right, front, and rear, walls relative to the front position. According to some examples, the room descriptor also may be provided in terms of a matrix, such as a 3x3 matrix. This room descriptor information is useful in describing the physical dimensions of the playback environment, for example in physical units of distance such as meters. In some such examples, sensory object locations, sensory object sizes, and sensory object orientations may be described in units that are relative to the room size, for example in a range from - 1 to 1.D24099W001Room descriptors may also describe a preferred viewing position, in some instances according to a matrix.

[0085] According to this example, system 500 includes a lightscape renderer 501 that is configured to render object-based light data 505 to light fixture control signals 515, based at least in part on environment and actuator data 104. In this example, the lightscape renderer 501 is configured to output the light fixture control signals 515 to light controllers 103, which are configured to control the light fixtures 108. The light fixtures 108 may include individual controllable light sources, groups of controllable light sources (such as controllable light strips), or combinations thereof. In some examples, the lightscape renderer 501 may be configured to manage various types of light object metadata layers, examples of which are provided herein. According to some examples, the lightscape renderer 501 may be configured to render actuator signals for light fixtures based, at least in part, on the perspective of a viewer. If the viewer is in a living room, that includes a television (TV) screen, the lightscape renderer 501 may, in some examples, be configured to render the actuator signals relative to the TV screen. However, in virtual reality (VR) use cases, the lightscape renderer 501 may be configured to render the actuator signals relative to the position and orientation of the user’s head. In some examples, the lightscape renderer 501 may receive input from the playback environment — such as light sensor data corresponding to ambient light, camera data corresponding to a person’ s location or orientation, etc. — to augment the render.

[0086] In some examples, the lightscape renderer 501 is configured to receive object-based light data 505 that includes light objects and object-based lighting metadata indicating an intended lighting environment, as well as environment and light fixture data 104 corresponding to light fixtures 108 and other features of a local playback environment, which may include, but are not limited to, reflective surfaces, windows, non-controllable light sources, light-occluding features, etc. In this example, the local playback environment includes one or more loudspeakers 109 and one or more display devices 510.

[0087] According to some examples, the lightscape renderer 501 is configured to calculate how to excite various controllable light fixtures 108 based at least in part on the object-based light data 505 and the environment and light fixture data 104. The environment and light fixture data 104 may, for example, indicate the geometric locations of the light fixtures 108 in the environment, light fixture type information, etc. In some examples, the lightscapeD24099W001renderer 501 may be configured to determine which light fixtures will be actuated based, at least in part, on the position metadata and size metadata associated with each light object, e.g., by determining which light fixtures are within a volume of a playback environment corresponding to the light object’s position and size at a particular time indicated by light object time stamp information. In this example, the lightscape renderer 501 is configured to send light fixture control signals 515 to the light controller 103 based on the environment and light fixture data 104 and the object-based light data 505. The light fixture control signals 515 may be sent via one or more of various transmission mechanisms, application program interfaces (APIs) and protocols. The protocols may, for example, include Hue API, LIFX API, DMX, Wi-Fi, Zigbee, Matter, Thread, Bluetooth Mesh, or other protocols.

[0088] In some examples, the lightscape renderer 501 may be configured to determine a drive level for each of the one or more controllable light sources that approximates a lighting environment intended by the author(s) of the object-based light data 505. According to some examples, the lightscape renderer 501 may be configured to output the drive level to at least one of the controllable light sources.

[0089] According to some examples, the lightscape renderer 501 may be configured to collapse one or more parts of the lighting fixture map according to the content metadata, user input (choosing a mode), limitations and / or configuration of the light fixtures, other factors, or combinations thereof. For example, the lightscape renderer 501 may be configured to render the same control signals to two or more different lights of a playback environment. In some such examples, two or more lights may be located close to one another. For example, two or more lights may be different lights of the same actuator, e.g., may be different bulbs within the same lamp. Rather than compute a very slightly different control signal for each light bulb, the lightscape renderer 501 may be configured to reduce the computational overhead, increase rendering speed, etc., by render the same control signals to two or more different, but closely-spaced, lights.

[0090] In some examples, the lightscape renderer 501 may be configured to spatially upmix the object-based light data 505. For example, if the object-based light data 505 was produced for a single plane, such as a horizontal plane, in some instances the lightscape renderer 501 may be configured to project light objects of the object-based light data 505 onto an upper hemispherical surface (e.g., above an actual or expected position of the user’s head) in order to enhance the experience.D24099W001

[0091] According to some examples, the lightscape renderer 501 may be configured to apply one or more thresholds, such as one or more spatial thresholds, one or more luminosity thresholds, etc., when rendering actuator control signals to light actuators of a playback environment. Such thresholds may, in some instances, prevent some light objects from causing the activation of some light fixtures.

[0092] Light objects may be used for various purposes, such as to set the ambience of the room, to give spatial information about characters or objects, to enhance special effects, to create a greater sense of interaction and immersion, to shift viewer attention, to punctuate the content, etc. Some such purposes may be expressed, at least in part, by a content creator according to sensory object metadata types and / or properties that are generally applicable to various types of sensory objects — such as object metadata indicating a sensory object’s location and size.

[0093] For example, the priority of sensory objects, including but not limited to light objects, may be indicated by sensory object priority metadata. In some such examples, sensory object priority metadata is taken into account when multiple sensor objects map to the same fixture(s) in a playback environment at the same time. Such priority may be indicated by light priority metadata. In some examples, priority may not need to be indicated via metadata. For example, the MS Renderer 001 may give priority to sensory objects — including but not limited to light objects — that are moving over sensory objects that are stationary.

[0094] A light object may, depending on its location and size and the locations of light fixtures within a playback environment — potentially cause the excitation of multiple lights. In some examples, when the size of a light object encompasses multiple lights, the renderer may apply one or more thresholds — such as one or more spatial thresholds or one or more luminosity thresholds — to gate objects from activating some encompassed lights.Examples of Using a Lighting Map

[0095] In some implementations a lighting map, which is an instance of the of the actuator map (AM) that includes a description of lighting in a playback environment, may be provided to the lightscape renderer 501. In some such examples, the environment and light fixture data shown in Figure 5 may include the lighting map. According to some examples, the lighting map may be allocentric, e.g., indicating absolute spatial coordinate-based light fall-off, whereas in other examples the lighting map may be egocentric, e.g., a light projectionD24099W001mapped onto a sphere at an intended viewing position and orientation. In the case of a sphere, the lighting map may, in some examples, be projected onto a two-dimensional (2D) surface, e.g., in order to utilize 2D image textures in processing. In any case, the lighting map should indicate the capabilities and the lighting setup of the playback environment, such as a room. In some embodiments the lighting map may not directly relate to physical room characteristics, for example if certain user preference-based adjustments have been made.

[0096] In some examples, there may be one lighting map per light fixture, or per light, in a playback environment. According to some examples, the intensity of light indicated by the light map may be inversely correlated to the distance to the center of the light, or may be approximately (e.g., within plus or minus 5%, within plus or minus 10%, within plus or minus 15%, within plus or minus 20%, etc.) inversely correlated to the distance to the center of the light. The intensity values of the light map may indicate the strength or impact of the light object onto the light fixture. For example, as a light object approaches a lightbulb, the lightscape renderer 501 may be configured to determine that the lightbulb intensity will increase as the distance between the light object and the lightbulb decreases. The lightscape renderer 501 may be configured to determine the rate of this transition based, at least in part, on the intensity of light indicated by the light map.

[0097] Inside this common rendering space, in some examples the lightscape renderer 501 may be configured to use a dot product multiplication between a light object and the light map for each light to compute a light activation metric, e.g., as follows:LM ■ ObjY~ min(£ LM, £ Obf)

[0098] In the foregoing equation, Y represents the light activation metric, LM represents the lighting map and Obj represents the map of a light object. The light activation metric indicates the relative light intensity for the actuator control signal output by the lightscape renderer 501 based on the overlap between the light object and the spread of light from the light fixture. In some examples, the lightscape renderer 501 may use the maximum or closest distance, or other geometric metrics, from the light object to the light fixture as part of the determination of light intensity. In some implementations, instead of computing the light activation metric, the lightscape renderer 501 may refer to a look-up-table to determine the light activation metric.D24099W001

[0099] The lightscape renderer 501 may repeat one of the foregoing procedures for determining the light activation metric for all light objects and all controllable lights of the playback environment. Thresholding for light objects that produce a very low impact on light fixtures may be helpful to reduce complexity. For example, if the effect of a light object would cause an activation of less than a threshold percent of light fixture activation — such as less than 10%, less than 5%, etc. — the lightscape renderer 501 may disregard the effect of that light object.

[0100] The lightscape renderer 501 may then use the resultant light activation matrix Y, along with various other properties such as the chosen panning law (either indicated by light object metadata or Tenderer configuration) or the priority of the light object, to determine which objects get rendered by which lights and how. Rendering lights-objects into light fixture control signals may involve:• Altering the luminance of a light-object as a function of the distance it is from the light fixture;• Mixing the colors of multiple light-objects that are simultaneously (multiplexed) rendered by a single light fixture; or• Altering either of the above based on the light object priority.RENDERING PARAMETERS

[0101] In addition to the information carried by the light object metadata, the rendering of light-objects can be a function of the settings or parameters of the lightscape renderer 501 itself. These may include:• Velocity priority - when this parameter is set, light objects that are moving are given a higher priority than those which are not. Having the velocity priority parameter set enhances the dynamism of the rendered scene;• Color priority - light-objects with higher saturation values will take priority;• Activation threshold - the minimum light activation, Y, that must be achieved in order to activate a light-fixture;• Accessibility - certain colors may be chosen over others to best represent the experience for colorblind users. Certain flash rates may be avoided for those with photo-sensitivities.D24099W001RENDERING CONFIGURATION (MODES)

[0102] In addition to the information carried by the light object metadata, the lightscape renderer 501 may, in some implementations, be configured according to different modes. As used herein, the term “mode” is different from “parameter” in the sense that modes may, for example, involve completely different signal paths, whereas parameters may simply parameterize these signal paths. For example, one mode may involve the projection of all light objects onto a lighting map before determining how / what to render to the light-fixtures, while another mode may only snap the highest-priority lights to the nearest light fixtures. Modes may include:• Modes to support low light- fixture count. In these modes, the rendering parameters and the light object metadata are utilized in order to determine which subset of light-objects are to be rendered and in what manner. Here, the “manner” refers to the trade-off between the spatial, color, temporal fidelity of the most prominent light-objects in the scene;• Modes to support different content types, such as music vs. gaming;• Modes in which multiple light objects may be rendered by a single light fixture (or a single light) with color mixing;• Modes in which only a single light object can be rendered by a single light fixture (or a single light);• Modes in which the luminance of the light object is altered as a function of the geometric - or otherwise - distance between the light object and light fixture.

[0103] As noted above, according to some implementations the system 500 may include one or more instances of the control system 110 of Figure 1 that are configured for performing at least some of the methods disclosed herein. In some such examples, one instance of the control system 110 may implement the lightscape creation tool 100 and another instance of the control system 110 may implement the experience player 002. In some examples, one instance of the control system 110 may implement the audio Renderer 006, the video Renderer 007, the lightscape renderer 501, or combinations thereof. According to some examples, an instance of the control system 110 that is configured to implement the experience player 002 may also be configured to implement the audio Renderer 006, the video Renderer 007, the lightscape renderer 501, or combinations thereof.D24099WO01

[0104] Figure 6 shows an example of a graphical user interface (GUI) that may be presented by a display device of the lightscape creation tool of Figure 5. As with other figures provided herein, the types and numbers of elements shown in Figure 6 are merely provided by way of example. Other GUIs presented by a lightscape creation tool may include more, fewer and / or different types and numbers of elements. According to some examples, the GUI 600 may be presented on a display device according to commands from an instance of the control system 110 of Figure 1 that is configured for implementing the lightscape creation tool 100 of Figure 5.

[0105] In this example, a user may interact with the GUI 600 in order to create light objects and to assign light object properties, which may be associated with the light object as metadata. According to this example, a user is selecting properties of the light object 630. In this example, the GUI 600 shows the light object 630 in a three-dimensional space 631, the latter of which represents a playback environment. Element 634 shows a coordinate system of the three-dimensional space 631. Accordingly, in this example the light object 630 and the three-dimensional space 631 are being viewed from the upper left.

[0106] A user may interact with the GUI 600 in order to select a position and a size of the light object 630. In some examples, a user may select a position of the light object 630 by dragging the light object 630 to a desired position within the three-dimensional space 631, for example by touching a touch screen, using a cursor, etc. According to some examples, a user may select a size of the light object 630 by selecting the size of the circle (or other shape) that is shown on the GUI 600 to indicate the outline of the light object 130. In some such examples, a user may decrease the size of the light object 630 via a two-fingered pinch of the outline of the light object 130, may increase the size of the light object 630 via a two- fingered expansion, etc.

[0107] Specifying the position and size of an MS object within an abstracted three-dimensional space, such as the three-dimensional space 631 of GUI 600, allows a content creator to generalize the position and extent of the corresponding MS effects without prior knowledge of the particular playback environment in which the MS effects will be provided. This is an advantage of the MS object-oriented approach of various disclosed implementations. For example, the GUI 600 allows a content creator to specify the position and size of the light object 630 within the three-dimensional space 631, thereby allowing the content creator to generalize the position and extent of the corresponding light effects,D24099W001without prior knowledge of the particular size of any particular playback environment in which the light effects will be provided, without prior knowledge of the number, type and positions of light fixtures, etc., within the playback environment in which the light effects will be provided, etc. The light fixtures that will potentially be actuated responsive to the presence of the light object 630 at a particular time will be those within a volume of the playback environment corresponding to the position and size / extent of the light object 630.

[0108] According to this example, a user may interact with the color circle 635 of the GUI 600 in order to select the hue and color saturation of the current light object and may interact with the slider 636 in order to select the brightness of the current light object. These and other selectable properties of the light object 630 are displayed in area 632 of the GUI 600. According to this example, the properties of the light object 630 that may be selected via the GUI 600 also include intensity, diffusivity, “feathering,” whether or not the light object is hidden, saturation, priority and layer. Light object layers and priority will be described in more detail below. Generally speaking, light object layers may be used to group light objects into categories such as “ambient,” “dynamic,” etc. Light object priority may be assigned by a content creator and used by a renderer to determine, for example, which light object(s) will be presented when two or more light objects are simultaneously active and are simultaneously encompassing an area that includes the same light fixture.

[0109] Area 640 of the GUI 600 indicates time information corresponding to each of a plurality of light objects that are being created via the lightscape creation tool. In this example, light objects are listed on the left side of the area 640, along a vertical axis, and time is shown along a horizontal axis. In this example, four-second time intervals are delineated by vertical lines. Here, time information for each light object is shown as isolated or connected diamond symbols or lines along a series of horizontal rows, each of which corresponds to one of the light objects indicated on the left side of the area 640. The line 633, for example, indicates that light object 3 will be displayed starting between 39 and 40 seconds and will be continuously displayed until almost 1 minute and 6 seconds. The diamond symbols to the right of the line 633 indicate that light object 3 will be displayed discontinuously for the next few seconds.

[0110] Figure 7 shows another example of a GUI that may be presented by a display device of the lightscape creation tool of Figure 5. As with other figures provided herein, the types and numbers of elements shown in Figure 7 are merely provided by way of example. OtherD24099W001GUIs presented by a lightscape creation tool may include more, fewer and / or different types and numbers of elements. According to some examples, the GUI 700 may be presented on a display device according to commands from an instance of the control system 110 of Figure 1 that is configured for implementing the lightscape creation tool 100 of Figure 5.

[0111] In this example, the GUI 700 represents an instant in time during which light fixtures in an actual playback environment are being controlled according to light objects that have been created by an implementation of the lightscape creation tool 100 of Figure 5. An image of the playback environment is shown in area 705 of the GUI 700. Various light fixtures 708 and a television 715 are shown in the playback environment of area 705. The particular instant in time is shown by vertical line 742 of area 740. At this time, the vertical line 742 intersects with horizontal lines 744a, 744b, 744c, and 744d, indicating that the light being provided in the corresponding light objects 1, 4, 5 and 7 are being played back. Area 732 indicates light object properties.

[0112] One may observe that at the instant in time that is depicted in Figure 7, the left side of the playback environment shown in area 705 is being illuminated by blue light. This corresponds, at least in part, to the effect of the light object 730 shown within the three-dimensional space 731.

[0113] According to this example, video data and audio data are also being played back in the audio environment, and the playback of rendered light objects is being synchronized with playback of the video data and audio data. In this example, an image of the played-back video is shown in area 710 of the GUI 700. The video may, for example, be played back by the television 715.

[0114] In some examples, a user may be able to interact with the GUI 700 in order to adjust light object properties, add or delete light objects, etc. For example, a user may cause the playback to be paused in order to adjust light object properties. In some alternative examples, a user may need to revert to a GUI such as the GUI 600 of Figure 6 in order to adjust light object properties, add or delete light objects, etc.

[0115] In the example described with reference to Figure 7, although the GUI 700 was being presented a display device corresponding to the lightscape creation tool 100 of Figure 5, light objects, audio and video were being rendered in an actual, real-world environment.Accordingly, in some implementations the example described with reference to Figure 7 mayD24099W001also involve at least some of the “downstream” rendering and playback functionality that can be provided by other blocks of Figure 5, including but not limited to that of the lightscape renderer 501, the light controller APIs 103—which may in some instances be implemented by the same device that implements the lightscape renderer 501—the light fixtures 108, the audio renderer 106, the loudspeakers 109, the video renderer 107 and the display device(s) 510. In some such examples, the processes described with reference to Figure 7 also may involve functionality of the experience player 102 of Figure 5.

[0116] In some alternative implementations, the example described with reference to Figure 7 may also involve at least some of the “downstream” rendering and playback functionality that can be provided by other blocks of Figure 4, including but not limited to that of the MS Renderer 001, the MS controller APIs 003 — which may in some instances be implemented by the same device that implements the MS Renderer 001 — the light fixtures 008, the audio Renderer 006, the loudspeakers 009, the video Renderer 007 and the display device(s) 010. In some such examples, the processes described with reference to Figure 7 also may involve functionality of the experience player 002 of Figure 4.

[0117] One goal of lightscape providers is to present an experience in the consumer playback environment that closely matches the intended experience as designed during lightbased content creation. Light-based content may also be referred to herein as lightscape content. Accordingly, some aspects of this disclosure outline systems and methods for color management at the creation, distribution and rendering stages of lightscapes. Devices and systems for providing such lightscape experiences should be capable of responding flexibly to the heterogeneities of each consumer playback environment. In this disclosure, methods of color management for consumer playback environments having heterogeneous lighting configurations are described. For the purposes of this disclosure, it will be assumed the lightbased content — such as light object data — includes desired color rendering information. Such information may, for example, be included in light object metadata that is provided along with corresponding light objects.

[0118] According to some examples, these systems and methods can provide an optimal rendering at a range of endpoint configuration types that best matches the creator’s intent. Some aspects of this disclosure involve implementing a color management workflow in the consumer playback environment — for example, implemented via a playback environment control system that includes a decoder and a Tenderer — to accurately play back and adapt theD24099W001light-based content. Some disclosed systems and methods are able to accommodate endpoints that have different color volumes than that in which the content was created, heterogenous light fixtures having different color volumes, TVs or other display screens having different color volumes than that of the light fixtures, or combinations thereof, while nonetheless displaying lightscape content congruently.Light Fixture Characterization

[0119] Light fixture characterization is important to the color management workflow. In simple terms, light fixture characterization involves defining the color volume of light sources, in other words how colorful and how bright the light sources are. In some examples, the process of light fixture characterization may be similar to current measurement techniques for televisions and may include measurement of the translation of a control signal to the light emitted from a light fixture or a component thereof. According to some examples, a color measurement device such as an integrating sphere or a colorimeter may be used to measure the light emitted from a light fixture or a component thereof during a light fixture characterization process. In some such examples, during the light fixture characterization process a signal generator may send a series of control signals to the light fixture to be measured and the color measurement device may measure the light emitted from the light fixture responsive to each of the control signals. Each control signal may be, or may correspond to, a code value. Each control signal may correspond to a different point within a color space. Accordingly, the light fixture characterization process may involve determining a transfer function, also referred to herein as a response function, corresponding to the response of the light fixture to each of the plurality of control signals.

[0120] According to some examples, the light fixture characterization process may involve one or more simplifying assumptions, which can lead to relatively more efficient, though potentially less accurate, light fixture characterization. In some such examples, a linear additive system may be assumed. Some such examples may involve measurement of a system “gamma” (relating code value response to optical power) and color primaries (defining a triangle of measurement). If, for example, a light fixture is a 3-primary system, as is sometimes the case with LED light fixtures, this type of light fixture characterization can work well. In some examples, a full 3D look-up-table that converts code value response (e.g., RGB, RGBW or RGBcWwW) to optical output may be used as part of the light fixture characterization process.D24099W001Color space definition

[0121] The optical output of the light sources and the desired reproducible color can be specified in many ways. Each way has unique advantages. For the purposes of this disclosure, any of the known color spaces may be used. Some common examples include xyY, XYZ, ICtCp. Whatever type of color representation is used, it should relate to physical, measurable, absolute lighting conditions. It can be helpful to match the content creation, distribution, and playback rendering color spaces so that it is not necessary to convert from one color space to another.

[0122] Various methods of color calibration for heterogeneous light fixtures are disclosed herein. Such methods may, for example, be performed by the control system 110 of Figure 1. In some examples, such methods may be performed by an instance of the lightscape renderer 501 or the light controller 103 of Figure 5.Absolute Calibration of Heterogenous Light Fixtures

[0123] The goal of absolute color management is to reproduce, as closely as possible given the particular light fixtures of a playback environment, the exact color specified in received light-based content, which is also referred to herein as lightscapes data. The lightscapes data may, for example, correspond to the decoded object-based light data 505 that the experience player 102 of Figure 5 provides to the lightscape renderer 501. As noted elsewhere herein, the light data 505 may include light objects and corresponding light metadata, which may include light object color metadata. In some examples, the exact color specified in the lightscapes data may be indicated by the light object color metadata. Some absolute color management methods may involve implementing a color volume mapping procedure that results in the xyY (or any other color representation) produced by each of the light fixtures in the playback environment being as close as possible to the xyY setpoint (or any other color representation) of the received light-based content and / or the light fixture control signals 515 produced by the lightscape renderer 501.

[0124] In principle, if the light fixtures are calibrated / measured and homogeneous, absolute color management could be accomplished easily. However, in practice, it is common for the playback environment to contain many heterogenous light fixtures. In these cases, it is common for the capabilities of these light fixtures to vary significantly. For example, the playback environment may have some light fixtures that are capable of reproducing a colorD24099W001and some that are not. This disclosure provides two general approaches to solving the problem:1. Aim for the least deviation from the target color; and / or2. Aim for the least deviation between light sourcesLeast Deviation from Target Color

[0125] In this category of solutions, the control system allows the light fixtures to deviate from a target color — for example, a target color that is indicated by the received light object color metadata — with the goal of achieving a close overall match to the target color. Some “least deviation” methods involve, for each light source, finding the closest match to the target color, e.g., according to one or more models of human vision. DeltaE (AE) is a family of algorithms specifically for calculating the difference (delta) between two colors, of which DeltaE ITP is a color difference metric designed to closely match human vision. Some disclosed “least deviation” methods involve using DeltaE_ITP in the minimization function between the target XYZ and the color volume of the light fixtures a playback environment. The color volume of the light fixtures the playback environment may be determined according to one of the above-described light fixture characterization processes. According to some such examples, the minimization function may be expressed as follows:min EITP(XYZTgt— XYZLight)

[0126] In the foregoing equation, EITPrepresents DeltaE_ITP, XYZTgtrepresents a position of a target color in a color space and XYZLightrepresents a position in a color space of a color produced by a light fixture. Other “least deviation” methods may involve using color difference metrics other than DeltaE_ITP in the minimization function, such as DeltaEIz, DeltaEITP, or another one of the DeltaE algorithms.Least Deviation Between Light Sources

[0127] In this category of solutions, the control system causes all light fixtures of the playback environment to produce the same color, as close to the target color as possible. In some such examples, in order to define a “common working space” for each light fixture of the playback environment, the control system finds the common intersection of the color volumes of each light fixture of the playback environment, which is also referred to herein as an overlapping light fixture color volume.D24099W001

[0128] Figure 8A shows an example of color volumes of two light fixtures of a playback environment. This example shows a light fixture color volume 860a for a first controllable light fixture and a light fixture color volume 860b for a second controllable light fixture within a color space 850. Figure 8A also shows an overlapping light fixture color volume 870, which is a volume of the color space 850 in which the light fixture color volume 860a and the light fixture color volume 860b overlap. Some “least deviation” methods may involve providing — for example, by control system that is implementing an instance of the lightscape renderer 501 or the light controller 103 of Figure 5 — light fixture control signals to the first controllable light fixture and the second controllable light fixture according to the overlapping light fixture color volume 870. In some such examples, all of the light fixture control signals will correspond to points in the color space 850 that are within the overlapping light fixture color volume 870. According to some examples, the control system may be configured to apply a model of human vision to find the reproducible color that is closest to the desired color within the overlapping light fixture color volume 870, e.g., according to one of the “least deviation” methods described above.

[0129] As with other figures provided herein, the types and numbers of elements shown in Figure 8A are merely provided by way of example. Other examples may include more, fewer and / or different types and numbers of elements. For example, some “least deviation” methods may involve evaluating color volumes of three or more light fixtures and evaluating an overlapping light fixture color volume for the three or more light fixtures. Some “least deviation” methods may involve providing, by a control system, light fixture control signals to each of the three or more light fixtures according to the overlapping light fixture color volume, e.g., such that all of the light fixture control signals correspond to points in the color space 850 that are within the overlapping light fixture color volume. In some examples, the control system may be configured to apply a model of human vision to find the reproducible color that is closest to the desired color within the overlapping light fixture color volume 870, e.g., according to one of the “least deviation” methods described above.

[0130] Color volume mapping often involves trade-offs when seeking to match a target color. With a limited color volume, in order to produce a color that is as close as possible to the target color, the control system may change luminance, hue and / or saturation. Each of these may potentially be changed in order to match the target color. Which of these attributes to emphasize, if any, may be a matter of personal preference. Allowing user input regarding luminance, hue and / or saturation preferences on the content creation side, the playback side,D24099W001or both, can lead to an enhanced experience. In some examples, user input regarding luminance, hue and / or saturation preferences may be obtained via one or more virtual sliders, one or more virtual knobs, etc., that are part of a graphical user interface (GUI) presented on a display.Just Noticeable Difference Model for Lightscapes

[0131] The previous section discusses methods for matching color / lighting characteristics of heterogeneous light fixtures that are not capable of reproducing a color identically, including methods that involve using a model of human vision. In this section, methods are disclosed that take into account additional environmental conditions that could affect the appearance of the rendered color. Here we introduce methods for estimating the “just noticeable difference” between light fixtures that are:• Separated by a significant distance;• Illuminating objects & walls etc. in between them; and / or• Not directly observable, for example when only an illuminated wall reflection is visible to the user.This section also describes how to use such models to improve the color calibration of heterogenous light fixtures in a playback environment.Separated by a Significant Distance

[0132] Consider a situation in which there are two light fixtures in a playback environment that should, according to a target color that is indicated by received light object color metadata, produce the same target color at the same time. It would be preferable to have the light produced by both of the light fixtures match the target color, and each other. The foregoing discussion provides two solutions to achieve a satisfactory match. However, the importance of the emitted light matching changes with the relative distance between the light fixtures. If the two light fixtures are located adjacent to each other, it is important that the colors of their emitted light match or else the difference will be easily observable. However, if the two light fixtures are located on opposite sides of the playback environment, it is less important that the colors of their emitted light match because the human visual system tolerance increases with the separation distance. Therefore, the required color-matchingD24099W001accuracy can be relatively less stringent if light fixtures are located relatively farther from one another.

[0133] This relationship between the required level of color-matching accuracy and light fixture proximity relates to the concept of “just-noticeable-difference” (JND). Accordingly, some disclosed color matching methods involve a model of lighting accuracy JND versus physical proximity. Some such methods are based in part on the following equation:min a · EITP(XYZTgt- XYZLight) + (1 - a) Σ EITP(XYZi- XYZj)i = lights, j = lightsIn the foregoing equation, a represents the distance between light fixtures i and j. In this example, the maximum value of a is 1. An a value of 1 could, for example, correspond to the entire x, y or z dimension of a playback environment, indicating that the light fixtures are located on opposite sides of the playback environment. When the light fixtures are far apart, a is 1 or close to 1, the first portion of the equation above (min a · EITP(XYZTgt— XYZLight)) is used. This is the same equation as described in the “Least Deviation from Target Color” section above. Because the light fixtures are far apart, applying the loss function causes each light source to produce light that is as close as possible to the target color.

[0134] The second half of the equation is describing a color difference match between all light sources, which in this example involves taking the sum of the color differences across all light sources. Therefore, when the light fixtures are in close proximity, a is close to zero and the loss function changes. In such instances, the control system needs to add a loss function component (the second half of the equation) that involves comparing the colors emitted by the light sources to each other.

[0135] In some alternative examples, instead of a summation here (which would allow one erroneous light source to be different) a maximizing operator may be implemented, which could work well to minimize across all light sources. This is a mathematical way to describe the “overlapping light fixture color volume” that is described above with reference to Figure 8A.D24099W001Illuminating Objects / Walls

[0136] When a light-fixture is illuminating another object within a playback environment, the reflectance or transmittance characteristics of the object affect the perception of the color being reproduced. For example, a tan lamp shade would reproduce white at a warmer temperature than a white shade. Therefore, in the color calibration calculation and when trying to match the colors from different light sources, it can be important to consider the reflection or transmittance characteristics of the surroundings. In some instances, a better match may be achieved by intentionally deviating from measured accuracy. In the case of the tan shade, a slightly blue-leaning lighting color would more accurately produce white.However, care must be taken if the light itself is also directly visible, because this causes a mixed color situation.Not directly observable

[0137] As in the previous example, let us suppose that a light fixture is illuminating one or more objects in a playback environment, but in this case, the light fixture itself is not visible. For example, a ceiling light may be illuminating a wall. The solution may be generally the same as noted in the previous example, but may be more flexible. In both cases (when a light source that is illuminating a reflecting surface is observable and when a light source that is illuminating a reflecting surface is not observable), there are perceptual components of “discounting the illuminant,’’ a process that helps the brain compensate for changes in illumination so that objects appear to maintain their color. However, when the light fixture is not visible it is more difficult for humans to distinguish light source color from reflectance color on a solid surface. Therefore, when the light fixture is not visible, there is more flexibility / opportunity to trick the mind into perceiving an intended reflected surface color. However, again care should be taken if the light source is illuminating objects with different reflectance properties.Relative Calibration of Heterogenous Light Fixtures

[0138] Some disclosed examples involve calibrating light fixtures such that they produce color that is most consistent across a set of light fixtures. For example, the control system may be configured to allow some deviation from the target xyY output of the renderer in order to reduce the deviation across an array of light fixtures. This may be very important for some low-cost light-emitting diode (LED) strips, such as those with 8 bit control and noD24099W001reliable gamma values. For object-based rendering, the output of the renderer is a function of the light fixture types, the light fixture locations and the light-based content itself. Thus, the target xy Y may not be deterministic for the content creator and small deviations from what the Tenderer computes may be acceptable. However, for channel-based content such as ambient beds, or for singular effects which actuate the entire playback environment (or large portions of it), then there is generally some deterministic expectation of the content creator: the content creator would normally expect the target color to be actually represented in the playback environment and the allowable deviation from the target xyY may be less or none at all. Some deviation may be permissible if, for example, there are also multiple overlapping light objects having colors that spatially mix and / or blend.Heterogenous Light Fixtures in the Presence of a Screen

[0139] This section involves combining absolute or relative calibration and the extended just noticeable difference model to optimally calibrate light fixtures across a playback environment. When a display screen, such as a television (TV) is present in the playback environment, it is generally very important to match the colors produced by that display screen with the colors produced by nearby light fixtures, especially if the goal of the lighting experience is to try to increase immersion in the scene by “extending the experience” of what is being presented on the display screen. In such examples, the display screen itself may be considered to be one of the light sources in the heterogenous system. Accordingly, the previously-described solutions are applicable. For example, a JND model may be based on the distance of light fixtures from a TV. Some examples may involve using overlapping color volumes of the light fixture(s) and the TV to calculate the available color palette. In some cases, changing at least some of the colors produced by the TV to match those produced by the light fixtures may be a viable solution that provides an improved user experience. These types of solutions may be desirable if the color volume produceable by the TV is larger than those of the light fixtures.

[0140] Figure 8B shows example elements of a system configured for receiving multi-sensory (MS) content and preparing the MS content for rendering in a playback environment. As with other figures provided herein, the types and numbers of elements shown in Figure 8B are merely provided by way of example. Other implementations may include more, fewer and / or different types and numbers of elements. According to some examples, system 800 may be, or may include, one or more devices configured for performing at least some of theD24099W001methods disclosed herein. In some examples, system 800 may include one or more instances of the control system 110 of Figure 1 that are configured for performing at least some of the methods disclosed herein. According to some examples, system 800 may include the MS Renderer 001 of Figure 4 or the lightscape renderer 501 of Figure 5.

[0141] According to this example, the system 800 includes a decoder 805 that is configured to receive and decode the encoded bitstream 801 and to provide the decoded bitstream 806 to blocks 810 and 870. In some examples, the encoded bitstream 801 may correspond to the object -based sensory data 005’ of Figure 4 and the decoded bitstream 806 may correspond to the object-based sensory data 005 of Figure 4. According to some examples, the encoded bitstream 801 may correspond to the object-based light data 505’ of Figure 5 and the decoded bitstream 806 may correspond to the object-based light data 505 of Figure 5. In some examples, the decoder 805 may be a component of the experience player 002 of Figure 4 or the experience player 102 of Figure 5.

[0142] In this example, the decoded bitstream 806 includes light objects 810, which are light objects in the ITU-R Recommendation BT.2020 (Rec2020) color space in this instance. Here, the light objects 810 have been quantized according to the perceptual quantizer (PQ) to 12 bits of precision (PQ ul2 Rec2020) objects in this example. The PQ may, for example, be the PQ published by Society of Motion Picture and Television Engineers (SMPTE) as SMPTE ST 2084. In other examples, the decoded bitstream 806 may include light objects having a different color space, a different quantization level, a different quantization type, etc.

[0143] According to this example, the transform block 815 is configured to transform the light objects 810 into light objects 817, which in this example have more bit depth than the light objects 810. In one example, the transform block 815 may be configured to transform the light objects 810 from unsigned 12-bit non-linear light objects into linear floating point data with more bit depth, such as 32-bit float (Lf32) light objects.

[0144] In some examples, the rendering module 820 may be configured to perform at least some of the functionality of the lightscape tenderer 501 of Figure 5. According to this example, the rendering module 820 is configured to produce light fixture control signals 824 based on the light objects 817. Accordingly, the rendering module 820 is configured to transform the light objects 817 from the object domain 812 to the light fixture domain 822.D24099W001The light fixture control signals 824 may, in some examples, be instances of the light fixture control signals 515 that are described with reference to Figure 5.

[0145] In this example, the transform block 825 is configured to transform the light fixture control signals 824 from a first color space to a second color space according to color space transformation parameters 852 received from the configuration data module 850 and to output transformed light fixture control signals 827 in the second color space. In some examples, the color space transformation parameters 852 may be “generic” parameters — in other words, known to those of ordinary skill in the art — for performing the color space transformation. According to some examples, the transform block 825 may be configured to transform the light fixture control signals 824 from the Rec 2020 color space to the ITP color space specified in the Rec. ITU-R BT.2100 standard and the color space transformation parameters 852 may indicate how to perform this color space transformation. The color space transformation performed by the transform block 825 may, in some examples, be the same for every light fixture in the playback environment. In some examples, the transform block 825 may be configured to perform a first pass or “coarse” color volume mapping process that involves this color space transform. The coarse color volume mapping process may, in some instances, involve data clipping. In this example, the coarse color volume mapping process is not specific to particular lights of the playback environment.

[0146] According to this example, the device-specific adjustment module 830 is configured for making fine adjustments to brightness, color volume mapping, etc., that are specific to particular lights and / or light arrays of the playback environment. Here, the device-specific adjustment module 830 includes the brightness scaling block 865 and the codeword generation block 875.

[0147] In this example, the brightness scaling block 865 is configured to perform what is referred to herein as a “brightness scaling” process on the transformed light fixture control signals 827, to produce the scaled light fixture control signals 866. Brightness scaling is a process in which the transformed light fixture control signals 827 are scaled according to various factors including perceptual brightness. In the example shown in Figure 8B, the brightness scaling process is based, in part, on a run-time brightness setpoint 862 that is set according to user input. In this example, the run-time brightness setpoint 862 is set by the user — potentially along with other runtime parameters — via a user input device that may include a GUI that includes a virtual brightness control (e.g., a brightness knob). AccordingD24099W001to some examples, the run-time brightness setpoint 862 may be a dynamic parameter that a user consuming the experience can change at any time. In some examples, the run-time brightness setpoint 862 may be the same for all light fixtures in the playback environment. According to some alternative examples, the run-time brightness setpoint 862 may be adjustable by a user on a per-zone basis: in other words, there may be a different run-time brightness setpoint 862 for each of two or more zones in the playback environment. For example, there may be one run-time brightness setpoint 862 for the front of a vehicle car and another run-time brightness setpoint 862 for the rear of the vehicle. According to this example, the brightness scaling process is also based, in part, on the a brightness tuning parameter 855 that is determined — in this example, along with other configuration data that is stored in the configuration data module 850 — when the playback system’s configuration is made and / or tuned. The brightness tuning parameter 855 allows the person tuning the playback system to make some lights dimmer or brighter according to their properties, installation, etc. For example, the brightness tuning parameter 855 may allow the person tuning the playback system to make the lights in a white vehicle somewhat dimmer, to make the lights that are expected to be directly pointing at a person dimmer, to scale the brightness of all the lights in the playback environment to similar levels, etc. According to some examples, the brightness tuning parameter 855 may be static and not adjustable by a user during a normal runtime or playback event, but may only be adjustable when the playback system’s configuration is made and / or tuned.

[0148] According to this example, the codeword generation block 875 is configured for determining and outputting codewords 880 corresponding to the scaled light fixture control signals 866 based, at least in part, on device-specific color volume parameters 856 received from the configuration data module 850 and rendering intent data 872 received from the rendering intent selection module 870. In some examples, the codewords 880 may correspond to the output signals from the light controller 103 of Figure 5, which are provided to the light fixtures 108 and used to control the colors, brightness, etc., of light generated by the light fixtures 108. The device-specific color volume parameters 856 may, in some examples, be determined as described above with reference to Figure 8A.

[0149] In some examples, the codeword generation block 875 may be configured to determine the codewords 880 based, at least in part, by reference to a data structure such as a look-up table (LUT). According to some such examples, the codeword generation block 875 may be configured to determine the codewords 880 by reference to a LUT for converting theD24099W001ITP color values of the scaled light fixture control signals 866 into color values of the codeword, which may RGB color values, RGBW color values, RGBcWwW color values, or other color values. The process of determining the codewords 880 may involve devicespecific — in other words, specific to a light fixture — clipping of the data according to the device-specific color volume parameters 856 and / or the rendering intent data 872. The process of determining the codewords 880 may, for example, be specific for sets of devices that share the same capability. For example, all light strips of type A might use one transform and all light strips of type B may use another transform.

[0150] According to this example, the rendering intent data 872 is obtained from the decoded bitstream 806, e.g., according to metadata in the decoded bitstream 806. In some examples, data clipping by the codeword generation block 875 may be based, at least in part, on the rendering intent metadata 872. For example, the rendering intent metadata 872 may indicate that if clipping occurs, whether hue or brightness with be prioritized. In some examples, the codeword generation block 875 is configured to change the direction that the ITP setpoint is moved onto the device’s color volume, based on the rendering intent metadata 872, before the codeword generation block 875 looks up the corresponding codeword. In some alternative examples, the codeword generation block 875 may perform data clipping after the transformation into codewords. However, the present inventors have found that this sequence of operations can sometimes yield sub-optimal results.

[0151] In this example, the device characterisation tool 833 is a tool that is configured to measure the responses of the light fixtures to be used in the playback environment and to provide device calibration data 834 to the device calibration library 835. Accordingly, the device calibration data 834 includes light fixture calibration data. In some implementations, the device characterisation tool 833 may be configured to measure the responses of other types of actuators in the playback environment, such as haptic devices, air flow devices, etc.

[0152] According to this example, the configuration generator 845 is configured to obtain device calibration data 834 from the device calibration library 835 and to obtain playback environment configuration data 840, the latter of which may include light fixture brightness tuning setpoints chosen by the tuner / user. In this example, the configuration generator 845 is configured to construct the coarse color space transform parameters 852 (e.g., for transforming from the Rec2020 RGB color space into the ITP color space) used by all light fixtures in the playback environment, as well as the device-specific color volume parametersD24099W001856 for producing the codewords 880. According to this example, the configuration generator 845 is configured to compute the brightness tuning coefficient used in the brightness scaling block 865 according to capabilities of the light fixtures in the configuration and all of the input brightness scaling parameters the tuner has provided.

[0153] Figure 9 is a flow diagram that outlines one example of a method that may be performed by an apparatus or system such as those disclosed herein. In this example, method 900 involves controlling a set of one or more controllable light fixtures of a playback environment. The blocks of method 900, like other methods described herein, are not necessarily performed in the order indicated. In some implementation, one or more of the blocks of method 900 may be performed concurrently. Moreover, some implementations of method 900 may include more or fewer blocks than shown and / or described. The blocks of method 900 may be performed by one or more devices, which may be (or may include) one or more instances of control system such as the control system 110 that is shown in Figure 1 and described above. For example, at least some aspects of method 900 may be performed by an instance of the control system 110 that is configured to implement the MS Renderer 001 of Figure 4 or the lightscape renderer 501 of Figure 5.

[0154] In this example, block 905 involves obtaining, by a control system, light fixture data for the set of one or more controllable light fixtures. In this example, the light fixture data includes light fixture color volume data for each controllable light fixture of the set of one or more controllable light fixtures.

[0155] According to this example, block 910 involves receiving, by the control system, light object data including a set of one or more light objects and corresponding light object metadata. In this example, the light object metadata includes intended light object color data and light object spatial data.

[0156] In this example, block 915 involves rendering, by the control system, the light object data to produce one or more light fixture control signals. In this example, the rendering is based at least in part on the light fixture color volume data and the light object metadata.

[0157] According to this example, block 920 involves providing, by the control system, the one or more light fixture control signals to one or more controllable light fixtures of the set of controllable light fixtures.D24099W001

[0158] In some examples, a first light fixture color volume for a first controllable light fixture of the set of one or more controllable light fixtures may differ from a second light fixture color volume for a second controllable light fixture of the set of one or more controllable light fixtures. In some such examples, the rendering may involve determining an overlapping light fixture color volume of the first light fixture color volume and the second light fixture color volume and providing the one or more light fixture control signals according to the overlapping light fixture color volume.

[0159] According to some examples, the rendering may be based at least in part on a viewer position, a viewer orientation, a distance between the first controllable light fixture and the second controllable light fixture, a distance between the first controllable light fixture and a display screen, a distance between the second controllable light fixture and the display screen, or combinations thereof.

[0160] In some examples, the rendering may be based at least in part on minimizing a difference between an intended light object color and a color produced by a controllable light fixture. According to some examples, the rendering may be based at least in part on minimizing a difference between a first color produced by a first controllable light fixture and a second color produced by a second controllable light fixture. In some examples, the rendering may be based at least in part on a just-noticeable-difference model.

[0161] In some examples, method 900 may involve obtaining, by the control system, display capability data for at least one display in the playback environment. According to some examples, the display capability data may include display color volume data. In some such examples, the rendering may be based at least in part on the display color volume data. According to some examples, the rendering may be based at least in part on minimizing a difference between a color produced by the display and a color produced by a controllable light fixture.

[0162] According to some examples, method 900 may involve determining, by the control system, an overlapping light fixture / display color volume between at least a first light fixture color volume and a display color volume and determining the one or more light fixture control signals according to the overlapping light fixture / display color volume. In some examples, method 900 may involve determining, by the control system, an overlapping light fixture / display color volume between at least a first light fixture color volume and a displayD24099W001color volume and controlling, by the control system, the display according to the overlapping light fixture / display color volume.

[0163] In some examples, method 900 may involve determining, by the control system, a modified white point based, at least in part, on minimizing differences in colors provided by each controllable light fixture of the set of one or more controllable light fixtures. In some such examples, the rendering may involve applying the modified white point. In some examples, determining the modified white point may be based, at least in part, on controlling one or more red / blue / green (RGB) outputs for at least one controllable light fixture to reduce color variation across the set of one or more controllable light fixtures.

[0164] According to some examples, method 900 may involve determining, by the control system, a color volume mapping procedure. In some such examples, the rendering may be based, at least in part, on the color volume mapping procedure. According to some examples, determining the color volume mapping procedure may involve minimizing a difference between an intended light object color coordinate in a color space and a light fixture color coordinate in the color space.

[0165] The above description illustrates various embodiments of the present disclosure along with examples of how aspects of the present disclosure may be implemented. The above examples and embodiments should not be deemed to be the only embodiments, and arc presented to illustrate the flexibility and advantages of the present disclosure as defined by the following claims. Based on the above disclosure and the following claims, other arrangements, embodiments, implementations and equivalents will be evident to those skilled in the art and may be employed without departing from the spirit and scope of the disclosure as defined by the claims.

[0166] Various aspects of the present disclosure may be appreciated from the following Enumerated Example Embodiments (EEEs):EEE1. A method of controlling a set of one or more controllable light fixtures of a playback environment, the method comprising:obtaining, by a control system, light fixture data for the set of one or more controllable light fixtures, the light fixture data including light fixture color volume data for each controllable light fixture of the set of one or more controllable light fixtures;receiving, by the control system, light object data including a set of one or more light objects and corresponding light object metadata, the light object metadata including intended light object color data and light object spatial data;D24099W001rendering, by the control system, the light object data to produce one or more light fixture control signals, wherein the rendering is based at least in part on the light fixture color volume data and the light object metadata; andproviding, by the control system, the one or more light fixture control signals to one or more controllable light fixtures of the set of controllable light fixtures.EEE2. The method of EEE1, wherein a first light fixture color volume for a first controllable light fixture of the set of one or more controllable light fixtures differs from a second light fixture color volume for a second controllable light fixture of the set of one or more controllable light fixtures and wherein the rendering comprises determining an overlapping light fixture color volume of the first light fixture color volume and the second light fixture color volume and providing the one or more light fixture control signals according to the overlapping light fixture color volume.EEE3. The method of EEE1 or EEE2, wherein the rendering is based at least in part on a viewer position, a viewer orientation, a distance between the first controllable light fixture and the second controllable light fixture, a distance between the first controllable light fixture and a display screen, a distance between the second controllable light fixture and the display screen, or combinations thereof.EEE4. The method of any one of EEE1-EEE3, wherein the rendering is based at least in part on minimizing a difference between an intended light object color and a color produced by a controllable light fixture.EEE5. The method of any one of EEE1-EEE3, wherein the rendering is based at least in part on minimizing a difference between a first color produced by a first controllable light fixture and a second color produced by a second controllable light fixture.EEE6. The method of any one of EEE1-EEE3, wherein the rendering is based at least in part on a just-noticeable-difference model.EEE7. The method of any one of EEE1-EEE6, further comprising obtaining, by the control system, display capability data for at least one display in the playback environment, the display capability data including display color volume data, wherein the rendering is based at least in part on the display color volume data.D24099W001EEE8. The method of EEE7, wherein the rendering is based at least in part on minimizing a difference between a color produced by the display and a color produced by a controllable light fixture.EEE9. The method of EEE7 or EEE8, further comprising determining, by the control system, an overlapping light fixture / display color volume between at least a first light fixture color volume and a display color volume and determining the one or more light fixture control signals according to the overlapping light fixture / display color volume.EEE10. The method of EEE7 or EEE8, further comprising determining, by the control system, an overlapping light fixture / display color volume between at least a first light fixture color volume and a display color volume and controlling, by the control system, the display according to the overlapping light fixture / display color volume.EEE11. The method of any one of EEE1-EEE10, further comprising determining, by the control system, a modified white point based, at least in part, on minimizing differences in colors provided by each controllable light fixture of the set of one or more controllable light fixtures, wherein the rendering involves applying the modified white point.EEE12. The method of EEE11, wherein determining the modified white point is based, at least in part, on controlling one or more red / blue / green (RGB) outputs for at least one controllable light fixture to reduce color variation across the set of one or more controllable light fixtures.EEE13. The method of any one of EEE1-EEE12, further comprising determining, by the control system, a color volume mapping procedure, wherein the rendering is based, at least in part, on the color volume mapping procedure.EEE14. The method of EEE13, wherein determining the color volume mapping procedure involves minimizing a difference between an intended light object color coordinate in a color space and a light fixture color coordinate in the color space.EEE15. An apparatus configured to perform the method of any one of EEE1-EEE14. EEE16. A system configured to perform the method of any one of EEE1-EEE14.D24099W001EEE17. One or more non-transitory and computer-readable media having instructions stored thereon to control one or more devices to perform the method of any one of EEE1-EEE14.

Claims

D24099W001CLAIMSWhat Is Claimed Is:

1. A method of controlling a set of one or more controllable light fixtures of a playback environment, the method comprising:obtaining, by a control system, light fixture data for the set of one or more controllable light fixtures, the light fixture data including light fixture color volume data for each controllable light fixture of the set of one or more controllable light fixtures;receiving, by the control system, light object data including a set of one or more light objects and corresponding light object metadata, the light object metadata including intended light object color data and light object spatial data;rendering, by the control system, the light object data to produce one or more light fixture control signals, wherein the rendering is based at least in part on the light fixture color volume data and the light object metadata; andproviding, by the control system, the one or more light fixture control signals to one or more controllable light fixtures of the set of controllable light fixtures.

2. The method of claim 1, wherein a first light fixture color volume for a first controllable light fixture of the set of one or more controllable light fixtures differs from a second light fixture color volume for a second controllable light fixture of the set of one or more controllable light fixtures and wherein the rendering comprises determining an overlapping light fixture color volume of the first light fixture color volume and the second light fixture color volume and providing the one or more light fixture control signals according to the overlapping light fixture color volume.

3. The method of claim 1 or claim 2, wherein the rendering is based at least in part on a viewer position, a viewer orientation, a distance between the first controllable light fixture and the second controllable light fixture, a distance between the first controllable light fixture and a display screen, a distance between the second controllable light fixture and the display screen, or combinations thereof.

4. The method of any one of claims 1-3, wherein the rendering is based at least in part on minimizing a difference between an intended light object color and a color produced by a controllable light fixture.D24099W0015. The method of any one of claims 1-3, wherein the rendering is based at least in part on minimizing a difference between a first color produced by a first controllable light fixture and a second color produced by a second controllable light fixture.

6. The method of any one of claims 1-3, wherein the rendering is based at least in part on a just-noticeable-difference model.

7. The method of any one of claims 1-6, further comprising obtaining, by the control system, display capability data for at least one display in the playback environment, the display capability data including display color volume data, wherein the rendering is based at least in part on the display color volume data.

8. The method of claim 7, wherein the rendering is based at least in part on minimizing a difference between a color produced by the display and a color produced by a controllable light fixture.

9. The method of claim 7 or claim 8, further comprising determining, by the control system, an overlapping light fixture / display color volume between at least a first light fixture color volume and a display color volume and determining the one or more light fixture control signals according to the overlapping light fixture / display color volume.

10. The method of claim 7 or claim 8, further comprising determining, by the control system, an overlapping light fixture / display color volume between at least a first light fixture color volume and a display color volume and controlling, by the control system, the display according to the overlapping light fixture / display color volume.

11. The method of any one of claims 1-10, further comprising determining, by the control system, a modified white point based, at least in part, on minimizing differences in colors provided by each controllable light fixture of the set of one or more controllable light fixtures, wherein the rendering involves applying the modified white point.

12. The method of claim 11, wherein determining the modified white point is based, at least in part, on controlling one or more red / blue / green (RGB) outputs for at least one controllable light fixture to reduce color variation across the set of one or more controllable light fixtures.D24099W00113. The method of any one of claims 1-12, further comprising determining, by the control system, a color volume mapping procedure, wherein the rendering is based, at least in part, on the color volume mapping procedure.

14. The method of claim 13, wherein determining the color volume mapping procedure involves minimizing a difference between an intended light object color coordinate in a color space and a light fixture color coordinate in the color space.

15. An apparatus configured to perform the method of any one of claims 1-14.

16. A system configured to perform the method of any one of claims 1-14.

17. One or more non-transitory and computer-readable media having instructions stored thereon to control one or more devices to perform the method of any one of claims 1-14.

Images