A controller for controlling operation of a lighting arrangement including one or more lighting devices located in an environment

Abstract

A controller for controlling operation of a lighting arrangement including one or more lighting devices located in an environment, the controller configured to ascertain a user input indicative of a lighting scene to be applied to the environment, ascertain lighting device data indicative of operational capabilities of the respective lighting devices. The controller is further configured to analyze the user input to determine that the user input is indicative of a dynamic lighting scene wherein at least one operation parameter of at least one of the lighting devices changes with time and generate a video based on the ascertained user input data and lighting device data. The controller is further configured to determine, based on the generated video, respective light settings indicative of lighting operation parameters for the one or more lighting devices, and to provide control signals CS indicative of the respective light settings for controlling the lighting devices

Description

[0001] 2024PF80326 1

[0002] A controller for controlling operation of a lighting arrangement including one or more lighting devices located in an environment

[0003] FIELD OF THE INVENTION

[0004] The present invention is directed to a controller for controlling operation of a lighting arrangement including one or more lighting devices located in an environment. The invention is further directed to a lighting arrangement including said controller and one or more lighting devices controllable by said controller. The invention is further directed to method for controlling operation of a lighting arrangement including one or more lighting devices located in an environment and a computer program.

[0005] BACKGROUND OF THE INVENTION

[0006] Modem lighting devices offer advanced control features, such as color control and dim level control. The light output from these devices can be either static, maintaining a constant colour and brightness, or dynamic, changing over time to create various effects. However, creating dynamic light effects can be cumbersome for users, as it often requires selecting a large number of colours and settings. This process can be time-consuming and complex.

[0007] To address this challenge, there is a growing need for simplified and intuitive control options that allow users to generate dynamic light effects without the hassle of manual adjustments. One promising approach to achieving this is through the use of natural language processing (NLP). By leveraging NLP, users can control lighting arrangements using simple voice commands or text inputs, making it easier to create and manage dynamic lighting effects. This technology has the potential to revolutionize the way we interact with lighting systems, making them more accessible and user-friendly.

[0008] SUMMARY OF THE INVENTION

[0009] It would be beneficial to provide a system that simplifies the creation of dynamic lighting scenes.

[0010] According to a first aspect of the invention, a controller is disclosed. The controller is configured to control operation of a lighting arrangement, which is an2024PF80326 2

[0011] arrangement that includes one or more lighting devices that are located in a given environment, such as a room, a warehouse, a street, a parking lot, etc.

[0012] The controller is configured to ascertain user input data that is correlatable (indicative of) to a lighting scene to be applied to the environment and lighting device data indicative of operational capabilities of the respective lighting devices. The controller is configured to analyze the user input to determine that the user input is indicative of a dynamic lighting scene wherein at least one operation parameter of at least one of the lighting devices changes with (as a function of) time. The controller is configured to generate a video based on the ascertained user input data and lighting device data. The controller is further configured to determine, based on the generated video, respective light settings indicative of lighting operation parameters for the one or more lighting devices, and to provide control signals indicative of the respective light settings for controlling the lighting devices.

[0013] According to the invention, the generation of the intermediate video from which lighting settings are extracted is conditioned on user input and on the operational capabilities of the different lighting devices. Thus, the determination of the light settings for the lighting device of the arrangement is based on user input data and tailored in dependence on the operational capabilities of the different lighting devices, as given by the, such that no lighting device is forced to operate under conditions that are not realistic or available for said lighting device (e.g., a color temperature of 2700K for a lighting device that has an operation range in terms of color temperature between 4000K and 5000K). The intermediate video acts as a kind of explainable- Al visualization for the user why the lighting control device has determined the applied light settings.

[0014] The user input data is interpreted as a suggestion, vague or explicit, of a particular lighting scene that is not necessarily pre-specified. A lighting scene is defined as a set of operation conditions of one or more lighting devices, wherein at least one of the lighting devices has controllable lighting parameters, for example in terms of, for example, colour spectrum, and / or light intensity, and / or colour temperature, and / or direction of illumination, and / or light beam forming, etc. The respective controllable parameters of the lighting devices are given by the lighting devices data. A lighting scene refers to the result of combining the different light settings applied to the different lighting devices. This enables a more realistic rendering of the user’s intent, as indicated by the user input, via the lighting units or lighting devices. Further it enables a more personalized use of smart lighting given the operational capabilities of the lighting devices.2024PF80326 3

[0015] A lighting scene can be a static scene, or a dynamic scene, wherein the operation parameters of at least one of the lighting device changes with time. The controller is advantageously configured to analyse the user input to determine that the user input is indicative of a dynamic lighting scene. By understanding the user's intent, the system can provide a more tailored lighting experience. Automatically distinguishing between static and dynamic scenes saves users from manually configuring complex settings. This makes the system more user-friendly and reduces the time and effort required to achieve the desired lighting effect.

[0016] In the following, embodiments of the lighting control device of the first aspect of the invention will be disclosed.

[0017] The controller may be configured to, based on the generated video, extract a sequence of color data (or color values) from subsequent segments or frames of the generated video, map the extracted sequence of color data to corresponding lighting operation parameters of the one or more lighting devices and determine the respective light settings based on the mapped lighting operation parameters. The color data includes information regarding the colors that are present in the video.

[0018] The controller may be further configured to ascertain environment data indicative of a spatial configuration of the environment and generate the video based on the ascertained user input data, environment data and lighting device data. The (intermediate) video which is then used to determine the light settings for the lighting devices, is generated based on the environment data, and thus depends on the spatial configuration of the environment wherein the lighting devices are placed. The spatial configuration of the environment is extracted from the ascertained environment data, which is preferably associated to a particular point-of-view of the user. This enables a more realistic rendering of the user’s intent, as the generation of the video is conditioned on the spatial layout of the environment where the lighting devices are operating.

[0019] The lighting device data can be for instance stored in a memory unit accessible by the lighting control device or integrated with it. The lighting device data pertaining to the lighting devices of the lighting devices can be transferred or generated during a commissioning process of the lighting arrangement. If new lighting devices are added to the arrangement the lighting device data is updated. Additionally, or alternatively, the lighting device data can be determined using the environment data, when the environment data is indicative of at least some of the lighting devices. For instance, environment data in the form of an image can include the image representation of one or more lighting devices of the2024PF80326 4

[0020] lighting arrangement, such as, for example, a wall sconce fixed to a wall and a pendant luminaire fixed to the ceiling. The lighting device data can be advantageously used to restrict which colors the generated video can include. For instance, lighting devices typically cannot generate brown light (as it is a subtractive color) and hence the controller is preferably configured not use browns in the generated video. Similarly, an RGB light cannot generate all the colors visible to the human eye.

[0021] In an embodiment, the user input comprises text and / or image and / or video and / or audio. For instance, the input user can include a text file including text data indicative of the user’s intent. Additionally, or alternatively, the user input can be an image file or a video file comprising image or video data from which a user intent can be extrapolated.

[0022] Further, the user input can comprise an audio file including audio data, showing also a user’s intent or mood related to which the lighting units or lighting devices are to be operated. For instance, a text file including the text “I am happy and I feel like flying” can be provided as user input data, and suitable light settings for the lighting devices will be provided based on said input. The user input data can be also an image of video file, for instance of a party at sunset by the beach, or an audio file, also indicative of a current mood or intent of the user upon which the lighting. The audio file can be converted into a text file or provided as is. The same applies to the image or video file. The user input may further comprise physiological data indicative of a psychological and / or physiological condition of the user. For example, EMG or EEG data, vital sign data indicative of a psychological and / or physiological condition of the user (e.g., happiness, stress, excitement, etc.).

[0023] Preferably, in an embodiment, the environment data is determined from an image of the environment (such as an RGB image, a depth map of a scanned environment using LiDAR, a neural radiance filed, an IR image, etc), which can be provided by the user.

[0024] The environment data may be indicative of object parameters indicative of object properties and / or object location of the objects located in the environment and / or delimiting the environment. The environment data can further comprise context awareness sensing data, for example indicative of how many people are present and / or where they are present in the environment, and, optionally also of a respective field-of-view. Additionally, or alternatively, the environment data can also be indicative of a current activity carried out by one or more of the people in the environment, or of an interaction between two or more people, such as fighting, hugging, etc. Here, the video is further generated based on the object parameters. The environment data can be obtained using image data or video data representing the environment. Alternatively, the environment data can be obtained using2024PF80326 5

[0025] audio data of text data describing the spatial configuration of the environment. Preferably, the image data is indicative of a current point-of-view of a user in the environment. The object parameters refer to properties and / or location of different objects in the environment or delimiting the environment. The objects can therefore include walls, ceilings, floors, furniture, doors, windows, plants.

[0026] The environment data may be indicative of object parameters that are indicative of object location of at least one of the lighting devices of the lighting arrangement and / or location of the light effect of tat least one of the lighting devices located in the environment, and wherein the video is generated based on the object parameters.

[0027] The object properties are preferably indicative of a colour, size, reflectivity, texture (e.g. hard / soft), material, etc., of the respective object. The object location refers to a location of the object in the environment and / or to a relative location between two or more objects. For instance, the environment data can be indicative of a ceiling height or a room surface. The environment data is preferably indicative of a colour distribution of the objects located in the environment. The environment data is preferably indicative of object parameters that are indicative of object location of at least one of the lighting devices of the lighting arrangement located in the environment, and wherein the video is generated based on the object parameters. In other words, in a preferred embodiment, at least one of the objects for which the object parameters are extracted is a lighting device of the lighting arrangement. The environment data is preferably indicative of the position of the lighting devices that are identifiable in the environment, for instance in an image of the environment, more particularly, a point-of-view image of the user.

[0028] Additionally, or alternatively, the environment data may be indicative of subject parameters indicative of subject properties and / or subject location of the subjects located in the environment. The subject properties can be based on vital sign or activity data indicative of the user and / or of the people located in the environment. The vital sign or activity data can be indicative of an emotional state of the subject (e.g., happy, sad, etc.), or sleepiness state of the subject.

[0029] Additionally, or alternatively, the environment data may comprise environmental parameters indicative of environmental conditions in the environment, including, but not limited to, a temperature in the environment, a humidity in the environment, air quality in the environment, weather conditions in the environment, a time of day, etc. The video is preferably generated using the object parameters and / or on the subject parameters, and / or environment parameters. For instance, if the environment parameters are2024PF80326 6

[0030] indicative of hot and humid conditions in the environment, the generated video can be generated to be indicative of a rainforest before a thunderstorm. The environmental conditions may comprise sound information pertaining to the environment, for example, audio signals indicative of a background music in the environment.

[0031] The controller may be configured to determine a motion pattern based on the user input. For example, natural language processing may be used to determine that inputs such as “sunset on the beach” involves the movement of the sun in the sky from east to west. The generation of the intermediate video may be conditioned on the determined motion pattern. In this way, the intermediary video captures the changes over time, maintaining the temporal consistency of objects, which is crucial for creating realistic dynamic scenes (e.g., leaves falling from a tree).

[0032] Preferably, in an embodiment, the controller is configured to generate the intermediate video using or based on a generative artificial intelligence (Al) or machine learning model. The controller may be configured to generate a prompt based on the user input, the lighting device data and optionally the environment data and to provide said prompt to the machine learning model configured to generate the video based on the prompt. The machine learning model may be based on a diffusion model or a GAN model or the decoder of a Variable Autoencoder Model.

[0033] The controller may be further configured to receive user feedback data indicative of a degree of satisfaction with the determined respective light-settings. The user feedback can be for instance in the form of a predetermined voice command or gesture or can be provided by interacting with a dedicated user interface, for instance an app on a mobile device. In this embodiment, the controller is configured to generate alternative video using user input data, lighting device data and optionally the environment data upon determining, based on the user feedback data that the degree of satisfaction is below a predetermined threshold. The controller can generate a completely new video or only re-generate portions of the video. For instance, if the feedback user is indicative of the fact the user liked a particular part or aspect of the video but was dissatisfied with another aspect, the controller can regenerate only that part of the video with which the user was not satisfied. Thus, if the user indicates that he or she is not satisfied with the light scene selected and implemented and / or with the video generated and displayed by providing a corresponding user feedback, the controller generates a subsequent video that is used to determine the corresponding light settings and to generate and provide the corresponding control signals.2024PF80326 7

[0034] A second aspect of the present invention is formed by a lighting arrangement. The lighting arrangement comprises a controller in accordance with the first aspect of the invention and one or more lighting devices that are configured to receive control signals indicative of the respective light settings and to operate in dependence thereof. Thus, the lighting arrangement of the second aspect of the invention shares the advantages of the controller of the first aspect.

[0035] The lighting control arrangement may further comprise a display device such as a smart phone, tablet, AR headset. The controller may be configured to display the generated video to the user via the display device.

[0036] A third aspect of the present invention is formed by a method for controlling operation of a lighting arrangement including one or more lighting devices located in an environment. The method of the third aspect comprises ascertaining user input that is indicative of a lighting scene to be applied to the environment, and lighting device data indicative of operational capabilities of the respective lighting devices. The method may further comprise ascertaining environment data indicative of a spatial configuration of the environment. The method also comprises analyzing the user input to determine that the user input is indicative of a dynamic lighting scene wherein at least one operation parameter of at least one of the lighting devices changes with time, and generating a video based on the ascertained user input data and lighting device data. The video generation may be further based on ascertained environment data. The method also comprises determining, based on the generated video, respective light settings indicative of lighting operation parameters for the one or more lighting devices, and providing control signals indicative of the respective light settings for controlling the lighting devices. Thus, the method of the third aspect shares the advantages of the lighting control device of the first aspect, or of any of its embodiments. In embodiments, the step of determining the respective light settings may comprises extracting a sequence of color data from subsequent segments of the generated video, mapping the extracted sequence of color data to corresponding lighting operation parameters for the one or more lighting devices, and determining the respective light settings based on the mapped lighting operation parameters.

[0037] According to a fourth aspect of the invention, a computer program comprising instruction which, when executed by a controller, cause the controller to perform the steps of the method according to the third aspect.

[0038] As will be appreciated by one skilled in the art, aspects of the present invention may be embodied as a device, a method or a computer program product.2024PF80326 8

[0039] Accordingly, aspects of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a "circuit", "module" or "system." Functions described in this disclosure may be implemented as an algorithm executed by a processor / microprocessor of a computer. Furthermore, aspects of the present invention may take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied, e.g., stored, thereon.

[0040] Any combination of one or more computer readable medium(s) may be utilized. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a computer readable storage medium may include, but are not limited to, the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of the present invention, a computer readable storage medium may be any tangible medium that can contain, or store, a program for use by or in connection with an instruction execution system, apparatus, or device.

[0041] A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

[0042] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber, cable, RF, etc., or any suitable combination of the foregoing. Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented2024PF80326 9

[0043] programming language such as Java(TM), Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0044] Aspects of the present invention are described below with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor, in particular a microprocessor or a central processing unit (CPU), of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer, other programmable data processing apparatus, or other devices create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0045] These computer program instructions may also be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function / act specified in the flowchart and / or block diagram block or blocks.

[0046] The computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0047] The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of devices, methods and computer2024PF80326 10

[0048] program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

[0049] BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The above, as well as additional objects, features and advantages of the disclosed systems, devices and methods will be better understood through the following illustrative and non-limiting detailed description of embodiments of devices and methods, with reference to the appended drawings, in which:

[0051] Fig. 1 shows a schematic block diagram of an embodiment of a lighting arrangement including an exemplary controller in accordance with the invention;

[0052] Fig. 2 shows a schematic block diagram of another embodiment of a lighting arrangement including another exemplary controller in accordance with the invention;

[0053] Fig. 3 shows schematically a method for controlling operation of a lighting arrangement in accordance with the invention.

[0054] All the figures are schematic, not necessarily to scale, and generally only show parts which are necessary in order to elucidate the invention, wherein other parts may be omitted or merely suggested.

[0055] DETAILED DESCRIPTION

[0056] Fig. 1 shows a schematic block diagram of an embodiment of a lighting arrangement 200 including an exemplary controller 100 in accordance with the invention. The controller 100 is configured to control operation of the lighting devices 202, 203, 204 located in an environment 206, for example, a living room. The controller 100 is configured to ascertain a user input or user input data UD that is correlatable to a lighting scene to be applied to the environment. The user input can for instance be a text, an image, a video, an2024PF80326 11

[0057] audio, etc and serves as a starting point for the generation of control signals for controlling operation of the lighting devices. It may be a direct indication of the user’s intention such as “I want to create a light scene based on a sunset” or “I want to create a light scene based on the following image” where the user input also includes the image file the user is referring to, or a more abstract text such as “I want to fly” from which a light scene can be inferred.

[0058] The controller 100 is further configured to ascertain lighting device data LD that is indicative of operational capabilities of the respective lighting devices 202, 203, 204. The lighting device data LD thus indicates how the different lighting devices can be operated, for instance in terms of intensity ranges, variable colour spectrum, colour temperature ranges, orientation or directivity of the emitted light, etc. For instance, lighting device 202 is a direct pendant luminaire with a LED light source that can emit white and colour light with a controllable intensity up to 1100 lumen and a colour temperature range between 2000K and 6500K. Lighting device 203 is a table top lamp with an LED light source that can emit white light with a controllable intensity up to 800 lumen and a colour temperature range from 2200K and 6500K. Lighting device 204 is a stand light with an LED light source that can emit white and colour light with a controllable intensity up to 470 lumen and a colour temperature range between 2000K and 6500K. The selection of the parameter values for operation of the lighting device is performed based on control signals that can be provided via a communication channel, which can be wireless or wired.

[0059] The controller 100 may be also configured to ascertain environment data ED indicative of a spatial configuration of the environment 206 where the lighting devices 202, 203, 204 are located. Preferably, the environment data ED is received as an image file 103 representing an image of the environment, for example an image of a point-of-view of a user. In particular, the environment data ED is indicative of object parameters indicative of object properties and / or object location of the objects located in the environment and / or delimiting the environment. For instance, in the environment shown in the image, there is a table 60 in the lower central part of the image. The environment data can also be indicative of spatial properties of the environment, such as, but not limited to, size, color and reflections of the walls, ceiling and / or floor of the environment, location of the lighting devices relative to each other and to the user, for instance when the image is a point-of-view image taken by the user.

[0060] The lighting device data LD can for instance be extracted from the environmental data, e.g., from the image of the environment, for example by performing an object recognition process, for instance based on a machine learning model, on the image data to identify the objects and classify them according to the respective operational2024PF80326 12

[0061] capabilities. Additionally, or alternatively, the lighting device data LD pertaining to the lighting devices 202, 203, 204 in the environment 206 can be stored in a memory unit 105 accessible by the controller 100. The controller 100 can use both the lighting device data stored in the memory unit 105 and the image 103 of the environment 206 represented by the environment data ED to determine which of the lighting devices are to be controlled. For instance, the memory unit can store lighting device data pertaining to lighting devices installed in different environments (kitchen, living room, master bedroom, etc.) and the actual set of lighting devices to be controlled or the given environment where the lighting devices to be controlled are installed can be extracted from the image 103 by analysing said image.

[0062] The controller 100 is further configured to analyze the user input to determine that the user input is indicative of a dynamic lighting scene wherein at least one operation parameter of at least one of the lighting devices changes with time. For example, natural language processing, such as based on a Large Language Model, may be used to infer context from the user input. The controller determines whether a static or a dynamic lighting scene is needed based on the inferred context. For example, if the user says, “I want a sunset effect”, the controller can interpret this as a request for a dynamic lighting scene that aims to mimic the natural progression of light during a sunset which involves changes in color and intensity, whereas if the user says, “I want to study”, the controller can interpret this as a request for a static lighting scene that provides steady and reliable lighting to support a focused activity such as lighting. In another example, if the user input is “overnight camping at the beach and enjoying the sunrise”, this is indicative of a time-series transition from a “moon-lit” lighting scene to “the sun rising from the ocean” lighting scene. This user input is hence comprised of at least two elements wherein each of the two elements is represented by a different lighting scene. Thus, the controller may analyze the user input to determine the presence of at least two elements indicative of different lighting scenes to determine that the user input is indicative of a dynamic lighting scene.

[0063] The controller 100 is configured to generate a video 107 based on the ascertained user input UD and lighting device data LD of the lighting devices that are to be controlled 202, 203, and 204. If, for instance, the lighting device 204 did not have the operational capability of rendering colors, based on the ascertained lighting device data LD, then the controller generates a video in accordance with the limited operational capability in terms of color rendering of the lighting device 204.

[0064] Preferably, the controller 100 is configured to generate the intermediate video based on the ascertained user input UD, the environment data ED and lighting device data2024PF80326 13

[0065] LD of the lighting devices that are to be controlled 202, 203, and 204. In particular, the video is further generated based on the object parameters of the objects located in the environment as extracted from the environment data. For example, the controller generates a video that includes a sun setting in the horizon, based on an exemplary “I need holidays” user input data and the image 103. The combination of the features in the generated video is based on the general concept of the user input, where the location and the colour of the features in the generated video 107 is correlated to the spatial configuration of the objects in the image 103 provided as environment data. For example, the generated video has prominent features at the locations of the lighting devices 202, 203, 204. For example, trajectory of the sun in at least some segments of the video positionally aligns with the locations of the lighting devices.

[0066] The controller is also configured to determine, based on the generated video, respective light settings indicative of lighting operation parameters for the one or more lighting devices, and to provide control signals CS indicative of the respective light settings for controlling the lighting devices 202, 203, 204. The light settings may specify how a colour palette and a distribution of the colours and colour temperatures among the available lighting devices 202, 203, 204 change over time.

[0067] Several methods and techniques are known for extracting light settings from a video. For example, the controller 100 may extract a sequence of color data from subsequent segments (frames) of the generated video. The colour data includes information regarding the colours that are present in the frames or segments of the generated video. The colour data may also include brightness data for each colour e.g. blue is has a brightness value and the red has a brightness value indicative of a fainter colour. In an embodiment, this is done using a dedicated module, such as a hue colour extraction module that is capable of extracting a colour palette from a video frame (an image). Colours from subsequent segments or frames of the generated video 107 may be extracted and mapped to lighting devices 202, 203, 204 by determining colours at locations in the corresponding video frame which correspond to the locations of the lighting device. When mapping colours from the frame of the generated video to lighting devices 202, 203, 204, it may be taken into account whether a lighting device is capable of rendering colour or white-only. Control signals indicative of the respective light settings are then provided to the lighting devices 202, 203, 204 for controlling their operation for rendering a light scene correlatable to the user’s intent as indicated in the user input UD and also based on the spatial configuration of the environment and on the capabilities of the respective lighting devices 202, 203, 204.2024PF80326 14

[0068] Fig. 2 shows a schematic block diagram of another embodiment of a lighting arrangement including another exemplary controller in accordance with the invention. This discussion will focus of the differences between the lighting arrangement 200 of Fig. 1 and the lighting arrangement 200 of Fig. 2. Those technical features having a similar or identical function will be referred to using the same reference signs and numbers.

[0069] In particular, the controller 100 of Fig. 2 is configured to generate a prompt P based on the user input and the lighting device data and to provide said prompt P to a machine learning model ML, for example configured as a text-to-video model, configured to generate the video based on the prompt. Preferably, the machine learning model ML is based on a diffusion model.

[0070] Thus, the controller 100 is configured to translate user’s intent for a lighting experience to rending the intended scene by generating an intermediary video that visually represent the intended lighting experience of the user from which subsequently light settings for a given environment are extracted and applied to the lighting devices of the lighting arrangement. The generation of the intermediary video is based on the user input and the lighting device data. In a preferred embodiment, the generation of the intermediary video is further based on the environment data.

[0071] Among the several algorithmic approaches to generate a video from the prompt P, which can be a natural language description or a multimodal prompt, a diffusion model approach is preferred. A foundational model (such as Stable Diffusion vl.4) is leveraged, and then its architecture can be modified and / or fine-tuned for this particular application. Preferably, contextual parameters are introduced. By using the contextual parameters, the diffusion process can be conditioned to generate a video that accurately captures the scene (e.g. running on the beach during sunset) that the user wants to use as a seed for extracting a new light effect to be rendered on the given environment of the user and the lights. A first approach is to train, e.g. from a supervised dataset, a conditional diffusion model to generate based on the user input a video which firstly represents the scene inspiring the user (e.g. running on the beach during sunset) and secondly is well suited to extract light settings for the users light from. If a labelled dataset is not available, synthetic data can be generated. Another conditioning method involves adapting pre-trained unconditional diffusion models to new conditions using the learned internal representations of the denoiser network. This approach is effective for various conditional generation tasks, including attribute-conditioned generation and mask-conditioned generation. By using the unconditional denoiser network as a feature extractor, guidance can be provided, which is2024PF80326 15

[0072] robust to the initial inaccurate estimates of xO (i.e., original input data), and the guidance directions can be quickly learnt from a small set of labelled samples. The denoiser network is trained with varying noise levels in its inputs and has learned to extract features of different scales at different time steps.

[0073] Considering all the information and data about the environment (location and properties of objects and lighting devices, such as material, colour, relative distance, orientation) and about the operational capabilities of the lighting devices (e.g., colour spectrum, colour temperature, intensity, directionality, etc.) the controller generates a prompt for the diffusion model and the intermediary video is generated.

[0074] In embodiments, the controller 100 may be configured to determine a motion pattern (or motion trajectory) based on the user input and generate the video based on the motion pattern using the machine learning model. In embodiments, the machine learning model may have been pre-trained conditioned on motion trajectories (or motion patterns). These trajectories can be sparse (few key points) or dense (many points), and they represent the motion of objects or the entire scene. The controller 100 then generates a motion prompt by defining the motion pattern for objects in the scene.

[0075] In an example, the user provided prompt as user input data states or is indicative of: “I am happy, and I feel like flying”. The user input e.g. “I feel like flying” is augmented for example with information about the location of the available light source for the user’s current field of view in the room and about the object properties of the objects in the environment. For instance, the prompt augmentation states: “a first light is a pendant luminaire located 1 m away from the user in northwestern direction at a height of 2.5 m above a white table. A second light is a color-tunable LED stand light and is located on 4m of the user in southern direction close to the wall. A third lighting device is a colourtemperature-tunable LED light located on the white table. The walls and the ceiling are white, and the floor has black and white tiles on a checkerboard pattern. Please generate a video which reflects the user’s emotion and has a motion trajectory of key features of the video at the location of the first, second and third light.” The prompt provided by the user as user input is provided alongside the augmentation as an augmented prompt P to the machine learning model such as the pre-trained diffusion model conditioned on motion patterns and the diffusion model generates the video 107 based on information related to the spatial configuration of the environment.

[0076] The lighting controller 100 may be configured to receive user feedback data indicative of a degree of satisfaction (or dissatisfaction) with the determined and2024PF80326 16

[0077] implemented light-settings and / or the generated video. In this exemplary controller, the controller is configured to generate alternative video using the user input UD, optionally the environment data ED and lighting device data LD upon determining, based on the user feedback data, that the degree of satisfaction is below a predetermined threshold. If the user is dissatisfied with the light settings generated based on the video 107 generated in Fig. 2, the user can express this dissatisfaction e.g., by interacting with a user feedback unit (e.g., voice activated, via a dedicated user interface, via an app, etc.) and the controller generates another video based on the same ascertained data.

[0078] Optionally, the lighting arrangement 200 of any of the Figs. 1 or 2 may comprise a display device 102. The display device can be, for instance a television, a smart phone, a tablet, a digital frame, a computer including a monitor, or any other device that display the corresponding video. The display device can also be an augmented reality (AR) headset configured to provide only a portion of the corresponding video to the user, which depends on the actual orientation of the user, as determined by the AR headset. Further, the controller can be configured to overlay one or more salient parts of features of the video onto the view of the environment. For instance, the generated video can include certain features (e.g., a setting sun), which can be overlaid onto the current view of the room or environment provided by the AR headset. In embodiments, the generated video may be overlaid on a depth map of the environment. The controller 100 may be configured to display (output) the corresponding generated video to the user via the display device, as shown exemplarily in Fig. 2. Optionally, the controller 100 is configured to display the video to the user in dependence on a predetermined content criterion and / or a predetermined resolution criterion. For instance, if the intermediary video 107 generated by the diffusion model is a very low-resolution not deemed to be human interpretable (though still useful for the colour palette picking algorithm), the controller can be configured to not display the video to the user via the display device. Further, depending on how this specific video must be refined before it can be shown to this user, the system decides how many diffusion steps can be skipped. If it is determined that the video is to be shown to the user, the controller can be configured to regenerate the video with a higher resolution (while for the purpose of extracting the colour data or determining the lighting settings for the environment, a low-resolution video may suffice). The generation of a higher resolution video can be triggered if the user challenges the automated lighting controls based on the low-resolution intermediate video.

[0079] Fig. 3 shows a flow diagram of an exemplary method 300 for controlling operation of a lighting arrangement 200 in accordance with the invention. The lighting2024PF80326 17

[0080] arrangement comprises one or more lighting devices located in an environment, such as a room. The method 300 comprises, in a step 302, ascertaining a user input that is correlatable to a lighting scene to be applied to the environment, and that serves as a starting point in the determination of the lighting scene. The user input is for example a text prompt, or an audio file, or an image or video file or physiological parameters of the user. The method may also comprise, in an optional step 304, ascertaining environment data indicative of a spatial configuration of the environment. The environment data is preferably indicative of object parameters that are indicative of object properties and / or object location of the objects located in the environment and / or delimiting the environment. The environment data is preferably provided as an image file. The method also comprises, in a step 306, ascertaining lighting device data indicative of operational capabilities of the respective lighting devices. The method further comprises, in a step 308, analyzing the user input to determine that the user input is indicative of a dynamic lighting scene wherein at least one operation parameter of at least one of the lighting devices changes with time. The method includes, in a step 310, generating image data based on the ascertained user input data and lighting device data. Step 308 may optionally comprise generating image data based on the ascertained user input data, environment data and lighting device data. This is optionally done by performing a step 309 for generating an augmented prompt based on the user input data, the environment data and the lighting device data and providing said augmented prompt to a machine learning model configured as a text-to-video model configured to generate the video based on the prompt, in particular a diffusion model. The method further includes, in a step 312, determining respective light settings indicative of lighting operation parameters for the one or more lighting devices based on the generated video, and, in a step 314, providing control signals indicative of the respective light settings for controlling the lighting devices.

[0081] The method 300 may be executed by computer program code of a computer program product when the computer program product is run on a processing unit of a computing device, such as the controller 100.

[0082] Various embodiments of the invention may be implemented as a program product for use with a computer system, where the program(s) of the program product define functions of the embodiments (including the methods described herein). In one embodiment, the program(s) can be contained on a variety of non-transitory computer-readable storage media, where, as used herein, the expression “non-transitory computer readable storage media” comprises all computer-readable media, with the sole exception being a transitory, propagating signal. In another embodiment, the program(s) can be contained on a variety of2024PF80326 18

[0083] transitory computer-readable storage media. Illustrative computer-readable storage media include, but are not limited to: (i) non-writable storage media (e.g., read-only memory devices within a computer such as CD-ROM disks readable by a CD-ROM drive, ROM chips or any type of solid-state non-volatile semiconductor memory) on which information is permanently stored; and (ii) writable storage media (e.g., flash memory, floppy disks within a diskette drive or hard-disk drive or any type of solid-state random-access semiconductor memory) on which alterable information is stored. The computer program may be run on the processor 302 described herein.

[0084] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0085] The corresponding structures, materials, acts, and equivalents of all means or step plus function elements in the claims below are intended to include any structure, material, or act for performing the function in combination with other claimed elements as specifically claimed. The description of embodiments of the present invention has been presented for purposes of illustration but is not intended to be exhaustive or limited to the implementations in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the present invention. The embodiments were chosen and described to best explain the principles and some practical applications of the present invention, and to enable others of ordinary skill in the art to understand the present invention for various embodiments with various modifications as are suited to the particular use contemplated.

Claims

2024PF80326 19CLAIMS:

1. A controller (100) for controlling operation of a lighting arrangement (200) including one or more lighting devices (202, 203, 204) located in an environment (206), the controller configured to:ascertain a user input (UD) indicative of a lighting scene to be applied to the environment;ascertain lighting device data (LD) indicative of operational capabilities of the respective lighting devices;analyze the user input to determine that the user input is indicative of a dynamic lighting scene wherein at least one operation parameter of at least one of the lighting devices changes with time;generate a video (107) based on the ascertained user input data and lighting device data; anddetermine, based on the generated video, respective light settings indicative of lighting operation parameters for the one or more lighting devices, and to provide control signals CS indicative of the respective light settings for controlling the lighting devices.

2. The controller according to claim 1, wherein determining the respective light settings comprises:extracting a sequence of color data from subsequent segments of the generated video;mapping the extracted sequence of color data to corresponding lighting operation parameters for the one or more lighting devices;determining the respective light settings based on the mapped lighting operation parameters.

3. The controller according to claim 1, wherein the controller is further configured to:ascertain environment data (ED) indicative of a spatial configuration of the environment;2024PF80326 20generate the video based on the environment data.

4. The controller of claim 2, wherein the environment data is determined from an image of the environment.

5. The controller of claim 2, wherein the environment data is indicative of object parameters that are indicative of object properties and / or object location of the objects located in the environment and / or delimiting the environment, and / or of subject parameters indicative of subject properties and / or subject location of the subjects located in the environment, and / or environmental parameters indicative of environmental conditions in the environment, and wherein the video is generated based on the object parameters and / or on the subject parameters, and / or the environment parameters.

6. The controller of claim 2, wherein the environment data is indicative of object parameters that are indicative of object location of at least one of the lighting devices of the lighting arrangement located in the environment, and wherein the video is generated based on the object parameters.

7. The controller of any of the preceding claims, wherein the controller is configured to determine a motion pattern based on the user input and generate the video based on the motion pattern.

8. The controller according to any of the preceding claims, wherein the user input comprises text and / or image and / or video and / or audio and / or physiological data indicative of a psychological and / or physiological condition of the user.

9. The controller of any of the preceding claims, wherein the controller is further configured to generate a prompt based on the user input data and the lighting device data and provide the prompt to a machine learning model configured to generate the video based on the prompt.

10. The controller of claim 9, wherein the machine learning model is based on a diffusion model or a GAN model or the decoder of a Variable Autoencoder Model.2024PF80326 2111. The controller of any of the preceding claims, further configured to receive user feedback data indicative of a degree of satisfaction with the determined respective lightsettings, and wherein the controller is configured to generate alternative video using the user input data and lighting device data upon determining, based on the user feedback data, that the degree of satisfaction is below a predetermined threshold.

12. A lighting arrangement (200) comprising:a controller (100) according to any of the preceding claims and one or more lighting devices (202, 203, 204) that are configured to receive control signals (CS) indicative of the respective light settings and to operate in dependence thereof.

13. The lighting arrangement of claim 12 further comprising a display device, wherein the controller is configured to display the video to the user via the display device.

14. A method (300) for controlling operation of a lighting arrangement including one or more lighting devices located in an environment, the method comprising:ascertaining (302) a user input indicative of a lighting scene to be applied to the environment;ascertaining (306) lighting device data indicative of operational capabilities of the respective lighting devices;analyzing (308) the user input to determine that the user input is indicative of a dynamic lighting scene wherein at least one operation parameter of at least one of the lighting devices changes with time;generating (310) a video based on the ascertained user input data and lighting device data; anddetermining (312), based on the generated video, respective light settings indicative of lighting operation parameters for the one or more lighting devices, and providing (314) control signals indicative of the respective light settings for controlling the lighting devices.

15. Computer program comprising instruction which, when executed by a controller, cause the controller to perform the steps of the method of claim 14.

Images