A depth sensing module, method and computer program for generating depth information for a scene

Abstract

A depth sensing module for generating depth information for a scene is provided, the depth sensing module comprising: an illuminator configured, once activated, to emit light of a first wavelength range within a first visual field to illuminate a portion of the scene; a sensor configured to detect light of the first wavelength range reflected from the scene; and a controller configured to generate depth information on the basis of the detected light and perform control to change an illuminated portion of the scene. A corresponding method and computer program are also provided.

Description

[0001] A DEPTH SENSING MODULE, METHOD AND COMPUTER PROGRAM FOR GENERATING DEPTH INFORMATION FOR A SCENE BACKGROUND

[0002] Field of the Disclosure

[0003] The present disclosure relates to a depth sensing module, method and computer program for generating depth information for a scene.

[0004] Description of the Related Art

[0005] The "background" description provided herein is for the purpose of generally presenting the context of the disclosure. Work of the presently named inventors, to the extent it is described in the background section, as well as aspects of the description which may not otherwise qualify as prior art at the time of filing, are neither expressly or impliedly admitted as prior art against the present invention.

[0006] Depth sensing is important in many different areas of technology. In particular, depth sensing is used in computer vision systems, in order to enable devices to determine the distance to objects or other components of a scene. For example, depth sensing systems may find use in robotics, inspection and monitoring, autonomous (including semi-autonomous) vehicle control systems or the like.

[0007] Depth sensing systems may include an illuminator and a sensor. The sensor may use reflected light from the scene in order to measure depth information.

[0008] In these depth sensing systems, there is often a trade-off between the measurable distance (i.e. the maximum distance from the depth sensor for which depth information can be obtained) and the field of view. Accordingly, the depth sensing system may not necessarily be able to provide depth information for all relevant parts of a scene. This can make it very difficult to provide accurate and reliable depth information for a scene.

[0009] This problem is exacerbated for a device with limitations or restrictions on possible power consumption, since the total area which can be illuminated by a illuminator is defined by the total power availability to that illuminator.

[0010] It is an aim of the present disclosure to address these issues.

[0011] SUMMARY

[0012] A brief summary about the present disclosure is provided hereinafter to provide a basic understanding related to certain aspects of the present disclosure.

[0013] Embodiments of the present disclosure are defined by the independent claims. Further aspects of the disclosure are defined by the dependent claims.

[0014] In accordance with the embodiments of the disclosure, improved measurement of depth information of a scene can be made.

[0015] The present disclosure is not particularly limited to these advantageous technical effects. Other technical effects will become apparent for the skilled person when reading the disclosure.

[0016] BRIEF DESCRIPTION OF THE DRAWINGSA more complete appreciation of the disclosure and many of the attendant advantages thereof will be readily obtained as the same becomes better understood by reference to the following detailed description when considered in connection with the accompanying drawings, wherein:

[0017] Figure 1 illustrates an example apparatus in accordance with embodiments of the disclosure;

[0018] Figure 2 illustrates an example depth sensing module in accordance with embodiments of the disclosure;

[0019] Figure 3 illustrates an example depth sensing module in accordance with embodiments of the disclosure;

[0020] Figure 4 illustrates an example depth sensing module in accordance with embodiments of the disclosure;

[0021] Figure 5 illustrates an example depth sensing module in accordance with embodiments of the disclosure;

[0022] Figure 6 illustrates an example depth sensing module in accordance with embodiments of the disclosure;

[0023] Figure 7 illustrates an example depth sensing module in accordance with embodiments of the disclosure;

[0024] Figure 8 illustrates an example depth sensing module in accordance with embodiments of the disclosure;

[0025] Figure 9 illustrates a sensor in accordance with embodiments of the disclosure;

[0026] Figure 10 illustrates an example method in accordance with embodiments of the disclosure;

[0027] Figure 11 illustrates an example implementation of a depth sensing module in accordance with embodiments of the disclosure;

[0028] Figure 12 illustrates an example method in accordance with embodiments of the disclosure.

[0029] DESCRIPTION OF THE EMBODIMENTS

[0030] Referring to Figure 1, an example apparatus 1000 (an example of an information processing device) according to embodiments of the disclosure is shown. Typical ly, an apparatus 1000 according to embodiments of the disclosure is a computer device such as a personal computer, an entertainment system or videogame console, or a terminal connected to a server. Indeed, in embodiments, the apparatus may also be a server. The apparatus 1000 is controlled using a microprocessor or other processing circuitry 1002. In some examples, the apparatus 1000 may be a portable computing device such as a mobile phone, laptop computer or tablet -computing device.

[0031] The processing circuitry 1002 may be a microprocessor carrying out computer instructions or may be an Application Specific Integrated Circuit. The computer instructions are stored on storage medium 1004 which maybe a magnetically readable medium, optically readable medium or solid state type circuitry. The storage medium 1004 may be integrated into the apparatus 1000 or may be separate to the apparatus 1000 and connected thereto using either a wired or wireless connection. The computer instructions may be embodied as computer software that contains computer readable code which, when loaded onto the processor circuitry 1002, configures the processor circuitry 1002 to perform a method according to embodiments of the disclosure.Additionally, an optional user input device 1006 is shown connected to the processing circuitry 1002. The user input device 1006 may be a touch screen or may be a mouse or stylist type input device. The user input device 1006 may also be a keyboard, controller, or any combination of these devices. In some examples, the user input device 1006 may be a microphone or other device. The user to may then provide input via sounds or speech.

[0032] A network connection 1008 may optionally be coupled to the processor circuitry 1002. The network connection 1008 may be a connection to a Local Area Network or a Wide Area Network such as the Internet or a Virtual Private Network or the like. The network connection 1008 may be connected to a server allowing the processor circuitry 1002 to communicate with another apparatus in order to obtain or provide relevant data. The network connection 1002 may be behind a firewall or some other form of network security.

[0033] Additionally, shown coupled to the processing circuitry 1002, is a display device 1010. The display device 1010, although shown integrated into the apparatus 1000, may additionally be separate to the apparatus 1000 and may be a monitor or some kind of device allowing the user to visualize the operation of the system (e.g. a display screen or a head mounted display). In addition, the display device 1010 may be a printer, projector or some other device allowing relevant information generated by the apparatus 1000 to be viewed by the user or by a third party.

[0034] As explained in the Background, depth sensing systems may include an illuminator and a sensor. The sensor may use reflected light from the scene in order to measure depth information.

[0035] In these depth sensing systems, there is often a trade-off between the measurable distance (i.e. the maximum distance from the depth sensor for which depth information can be obtained) and the field of view. For example, when longer-range measurements are required, a depth sensor with a narrow field of view may be used. However, this does not enable the depth sensing system to perform measurements of a distance that is outside of the field of view.

[0036] Accordingly, the depth sensing system may not necessarily be able to provide depth information for all relevant parts of a scene. This can make it very difficult to provide accurate and reliable depth information for a scene. As such, improved measurement of depth information of a scene can be made since depth information can be generated for all relevant portions of the scene.

[0037] For at least these reasons, a depth sensing module, method and computer program are provided in accordance with embodiments of the disclosure.

[0038] <Depth Sensing Module>

[0039] Consider, now, Figure 2 of the present disclosure. Figure 2 illustrates an example depth sensing module 2000 in accordance with embodiments of the disclosure.

[0040] The example depth sensing module 2000 comprises an illuminator 2002, a sensor 2004 and a controller 2006.

[0041] The illuminator 2002 is configured, once activated, to emit light of a first wavelength range within a first visual field to illuminate a portion of the scene.

[0042] The sensor 2004 is configured to detect light of the first wavelength range reflected from the scene. Finally, the controller 2006 is configured to generate depth information on the basis of the detected light and perform control to change the illuminated portion of the scene.In this way, the depth sensing module 2000 is able to obtain depth information not only for a portion of the scene which falls within the visual field of the illuminator, but can also perform control to change the illuminated portion of the scene, such that depth information can be obtained from a portion of the scene which (before the control was performed) was located outside the visual field of the illuminator.

[0043] It will be appreciated that the example of Figure 2 is merely one example of a depth sensing system in accordance with embodiments of the disclosure. A number of additional components may, optionally, be provided as part of the depth sensing system as required. As an example, the depth sensing module 2000 of Figure 2 may, optionally, also comprise a detector 2008. The detector 2008 may be configured to detect one or more conditions, with the controller being configured to perform control to change the illuminated portion of the scene in accordance with the one or more detected conditions. This will be described in more detail later.

[0044] In examples, the light emitted by the illuminator (i.e. the light of the first wavelength range) may be light of a non-visible wavelength range. For example, the light may include light of an infrared wavelength range. In examples, the infrared wavelength range of the light may be a wavelength range of light between a range of 750 nanometres to 1550 nanometres. However, the present disclosure is not particularly limited in this regard and may, more generally, include light of any wavelength range.

[0045] In examples, the illuminator 2002 may include a illuminator such as an Light Emitting Diode (LED). In examples, the illuminator may include an illuminator such as a laser. In examples, the illuminator may be part of an illuminator array. As an example, for a depth sensing system such as a Light Detection and Ranging system (LiDAR), the illumination array may be a laser array. In examples, the laser array may be a solid-state laser array. Indeed, in the example of a LiDAR system, the illumination array may include a vertical activity surface emitting laser (VCSEL) array formed of a plurality of individual lasers.

[0046] However, the type of illuminator used is not particularly limited, provided that it can emit light of the first wavelength range for use in the depth sensor.

[0047] Light emitted from the illuminator illuminates a portion of the scene, being the portion of the scene within the visual field of the illuminator (i.e. the illuminated portion of the scene is defined by the visual field of the illuminator). The visual field of the illuminator is defined as the area covered by the field of view of the illuminator (the angular extent of the scene which can be illuminated) and the range of the light emitted from the illuminator (the maximum distance from the sensing module the illuminator can illuminate with sufficient illumination in order for depth measurement to be performed).

[0048] The light emitted from the illuminator reflects from the illuminated portion of the scene (e.g. it reflects off the surface or objects within that illuminated portion of the scene) and is received by the sensor of the depth sensing module. Thus, the sensor can detect the light of the first wavelength range reflected form the scene, with the detection performed by the sensor being used in order to generate depth information for the scene (in particular, the portion of the scene illuminated by the illuminator, from which the reflected light is thus received).

[0049] Indeed, in a depth sensing system which utilizes time-of-flight information (such as a LiDAR system), depth information is obtained by measuring the time taken for reflected light emitted by the illumination array to return to the sensor. Therefore, the illumination array 2000 may rapidly pulse to produce the illumination which can then be used in order to "scan" the scene (and thus producedepth information). In this way, a depth sensing system (such as a LiDAR system) may produce illumination (such as laser illumination) which can be used in order to scan the scene and which can be used, when reflected off objects in the scene, to determine depth information concerning those objects within the scene.

[0050] The type of sensor 2004 in the depth sensing module is not particularly limited in the present disclosure, provided that the sensor is able to detect the light of the first wavelength range reflected from the illuminated portion of the scene. Further details of the sensor 2004 will be described later with reference to Figure 10 of the present disclosure.

[0051] In examples, the pixels of the image sensor may be depth sensing pixels configured to acquire depth information based on time-of-flight information.

[0052] In examples, the pixels of the image sensor 2002 may comprise single photon detectors (i.e. pixels which can detect arrival of a single photon). In examples, the pixels of the image sensor 2002 may include single photon avalanche diode detectors (SPADs). A SPAD comprises a photodiode with a reverse bias voltage higher than the breakdown voltage of the photodiode. A single photon incident on the SPAD can trigger a self-sustaining avalanche. Therefore, the SPAD is very sensitive with the ability to detect a single photon.

[0053] However, it will be appreciated that the present disclosure is not particularly limited in this regard. Any other type of pixel may be used in the image sensor of the depth sensing system as required. As previously noted, there is a trade of between the field of view of the illuminator and the range at which the illuminator can illuminate the scene. That is, illuminators with narrower field of view (i.e. narrower angular extent of illumination) can measure a longer distance without increasing the power needed for the illumination (as the power required for illumination depends on the visual field itself (that is, it depends on the area of the visual field of the illuminator)).

[0054] The depth sensing module of the present disclosure is able to perform control to change the illuminated portion of the scene (i.e. the portion of the scene which falls within the visual field of the illuminator). In other words, the visual field of the illuminator is not fixed, but rather can be controlled by the controller of the depth sensing module. This means that depth information can be obtained for all relevant portions of the scene (even if those portions of the scene were, originally, located outside the visual field of the illuminator).

[0055] In examples, the controller 2006 may be implemented as processing circuitry such as the processing circuitry 1002 as described with reference to Figure 2 of the present disclosure. In the example of Figure 2, the controller 2006 is shown separate from the sensor 2004. However, in examples, the controller may be combined or integrated with the sensor. As such, the present disclosure is not particularly limited to the configuration illustrated in the example of Figure 2.

[0056] Further details of embodiments of the disclosure will now be described with reference to a number of additional example configurations of a depth sensing module of the present disclosure.

[0057] <Actuation>

[0058] Consider, now, Figure 3 of the present disclosure. Figure 3 of the present disclosure illustrates an example depth sensing module in accordance with embodiments of the disclosure.

[0059] The depth sensing module of Figure 3 of the present disclosure is similar to the depth sensing module which has been described with reference to Figure 2 of the present disclosure, in that it comprises an illuminator, a sensor 2004 and a controller 2006. Furthermore, similar to the depthsensing module of Figure 2, the depth sensing module of Figure 3 may also, optionally, comprise a detector 2008.

[0060] Therefore, further discussion of the features of the depth sensing module illustrated in the example of Figure 3 which are the same as those features which have already been described in the example of Figure 2 will not be provided again at this stage.

[0061] However, in this specific example of Figure 3, the illuminator 3002 is an illuminator which includes, or is connected to, an actuation element (not, in itself, shown in Figure 3). The actuation element is any element which can cause actuation of the illuminator 3002, in order to change one or a position and / or orientation of the illuminator 3002 (that is, any element which can be used in order to change the currently illuminated portion of the scene).

[0062] The type of actuating element used in order to provide actuation of the illuminator 3002 is not particularly limited in accordance with embodiments of the disclosure. For example, the actuating element may be an alternating current motor, a direct current motor, a linear motor, a stepper motor, an induction motor, a piezoelectric motor, or the like. Indeed, any suitable actuation element can be used in accordance with embodiments of the disclosure, provided that, through its actuation, it is able to control at least one of a position and / or orientation of the illuminator 3002.

[0063] In this example, the actuation element is controlled by the controller 2006. That is, the controller 2006 is configured to perform control to change the illuminated portion of the scene by controlling actuation of the illuminator.

[0064] Consider, for example, a situation where the illuminator is configured, once activated, to emit light of a first wavelength range within a first visual field to illuminate a portion of the scene. The visual field in which the illuminator can emit light may be defined by the region A as illustrated in Figure 3 of the present disclosure. In other words, this region A as illustrated in Figure 3 forms the portion of the scene which can be illuminated by the illuminator and, thus, the portion of the scene for which depth information (based on a detection of reflected light from the scene) can be generated.

[0065] However, this region A forms only a portion of the total scene (the region S in Figure 3). Therefore, based on the current visual field illuminated by the illuminator, the depth sensing module would not be able to generate depth information for any portion of the scene outside this region A. However, in accordance with embodiments of the disclosure, the controller 2006 is able to perform control to change the illuminated portion of the scene. In particular, in this example, the controller is able to perform control to change the illuminated portion of the scene by controlling actuation of the illuminator 3002.

[0066] For example, the controller 2006 may control actuation of the illuminator 3002 such that the portion of the scene S illuminated by the illuminator changes from the portion A to the portion B illustrated in Figure 3 of the present disclosure. In this way, depth information can be generated, by detection of reflected light from the scene, from region B - even though this region B was originally located outside the portion A of the scene illuminated by the illuminator.

[0067] In examples, the controller may control the actuation of the illuminator such that the illuminator illuminates each portion of the scene in turn (e.g. through a sweeping pattern). Therefore, even though only two regions of the scene (regions A and B) are shown as illuminated by the illuminator in Figure 3, it will be appreciated that the present disclosure is not particularly limited in this respect. However, the entirety of the scene does not necessarily have to be illuminated by the illuminator: in examples, the controller may perform control to change the illuminated portion of the scene bycontrolling actuation of the illuminator 3002 such that a certain target region (e.g. a region of interest) is illuminated. This enables the depth sensing module to acquire depth sensing information of any relevant section of the scene as required. Specific targeting of a region of interest of the scene may be based on the one or more conditions detected by the detector (if present) and will be discussed in more detail later.

[0068] Therefore, it will be appreciated that even though the illuminator 3002 is able to illuminate only a portion of the scene, the depth sensing module can still generate depth information for any relevant section of the scene through control to change the illuminated portion of the scene. Furthermore, it will be appreciated that, in some examples, the control to change the illuminated portion of the scene can be performed by controlling actuation of the illuminator 3002.

[0069] <Power>

[0070] Consider, now, Figure 4 of the present disclosure. Figure 4 of the present disclosure illustrates an example depth sensing module in accordance with embodiments of the disclosure.

[0071] The depth sensing module of Figure 4 of the present disclosure is similar to the depth sensing module which has been described with reference to Figure 2 of the present disclosure, in that it comprises an illuminator, a sensor 2004 and a controller 2006. Furthermore, similar to the depth sensing module of Figure 2, the depth sensing module of Figure 3 may also, optionally, comprise a detector 2008.

[0072] Therefore, further discussion of the features of the depth sensing module illustrated in the example of Figure 4 which are the same as those features which have already been described in the example of Figure 2 will not be provided again at this stage.

[0073] However, in this specific example of Figure 4, the illuminator is an illuminator which is configured to provide a different field of illumination depending on a level of power which is supplied to the illuminator. Accordingly, in this example, the controller is configured to perform control to change an illuminated portion of the scene by controlling a level of power supplied to the illuminator 4002. In general, the power which may be supplied to the illuminator may be limited depending on the type of the device in which the depth sensing module is located. For example, in a mobile or portable device, the power which may be supplied to the illuminator may be limited depending on a type of battery (or a capacity of a battery) which supplies power to the depth sensing module. In other examples, the power which may be supplied to the illuminator may be limited based on a level of heat generated by the illuminator. That is, in some examples, increasing the power which is supplied to the illuminator may increase the amount of heat which is dissipated by the illuminator. Accordingly, it may be necessary to restrict the level of power supplied to the illuminator in order in accordance with the availability of power resources (e.g. remaining capacity of a battery) and / or a limit on an amount of heat which can be dissipated by the illuminator.

[0074] The total area which can be illuminated by the illuminator with sufficient illumination in order to perform depth measurement (i.e. the visual field of the illuminator) is defined by the level of power which can be supplied to that illuminator. An increase in the visual field of the illuminator (by increasing either the range of the illuminator and / or the field of view of the illuminator) can only be achieved if the level of power supplied to that illuminator increases. On the other hand, if the level of power supplied to the illuminator decreases, then the visual field of that illuminator must also decrease (either by decreasing the range and / or the field of view of the illuminator).However, in some situations, it will be appreciated that a restriction may be placed upon the average amount of power which can be supplied to the illuminator. For example, it may be that the rate of power consumption of the illuminator over a certain period of time should remain within a certain predetermined limit. Nevertheless, despite this restriction, it may be possible to temporarily exceed the restriction placed on the amount of power which can be supplied to the illuminator (e.g. either such that, over a period of time, the average amount of power supplied to the illuminator remains within a certain predetermined limited or, alternatively, such that the restriction on the amount of power supplied to the illuminator exceeds the restriction on the amount of power for only a limited time). In other words, the controller may, in certain situations, be able to perform control to supply a boost in power supplied to the illuminator which, in turn, can be used in order to increase the visual field of the illuminator (and thus to illuminate an increased portion of the scene). Consider, for example, a situation where the illuminator is configured, once activated, to emit light of a first wavelength range within a first visual field to illuminate a portion of the scene. The visual field in which the illuminator can emit light may be defined by the region Al as illustrated in Figure 4 of the present disclosure. In other words, this region Al as illustrated in Figure 4 forms the portion of the scene which can be illuminated by the illuminator and, thus, the portion of the scene for which depth information (based on a detection of reflected light from the scene) can be generated.

[0075] However, this region Al forms only a portion of the total scene (the region S in Figure 4). Therefore, based on the current visual field illuminated by the illuminator, the depth sensing module would not be able to generate depth information for any portion of the scene outside this region Al. However, in accordance with embodiments of the disclosure, the controller 2006 is able to perform control to change the illuminated portion of the scene. In particular, in this example, the controller is able to perform control to change the illuminated portion of the scene by controlling a level of power supplied to the illuminator 4002.

[0076] For example, the controller 2006 may perform control to increase the power supplied to the illuminator 4002 such that the portion of the scene S illuminated by the illuminator changes from the portion Al to the portion Bl illustrated in Figure 4 of the present disclosure. In this way, depth information can be generated, by detection of reflected light from the scene, from region Bl - even though this region Bl was originally located outside the portion Al of the scene illuminated by the illuminator.

[0077] This enables the depth sensing module to acquire depth sensing information of any relevant section of the scene as required. Specific targeting of a certain region of the scene, such as a region of interest, may be based on the one or more conditions detected by the detector (if present) and will be discussed in more detail later. Nevertheless, while only two distinct regions of the scene are shown as being illuminated in the example of Figure 4 (namely regions Al and Bl) it will be appreciated that the present disclosure is not particularly limited in this regard. More generally, control may be performed such that any suitable region of the scene is illuminated as required depending on the situation to which the embodiments are applied.

[0078] Therefore, it will be appreciated that even though the illuminator 4002 is able to illuminate only a portion of the scene, the depth sensing module can still generate depth information for a relevant section of the scene through control to change the illuminated portion of the scene. Furthermore, it will be appreciated that, in some examples, the control to change the illuminated portion of the scene can be performed by controlling the level of power which is supplied to the illuminator.Moreover, in some examples, control of the actuation of the illuminator as described with reference to Figure 3 of the present disclosure can be combined with control of the power supplied to the illuminator.

[0079] <Optical Element>

[0080] Consider, now, Figure 5 of the present disclosure. Figure 5 of the present disclosure illustrates an example depth sensing module in accordance with embodiments of the disclosure.

[0081] The depth sensing module of Figure 5 of the present disclosure is similar to the depth sensing module which has been described with reference to Figure 2 of the present disclosure, in that it comprises an illuminator, a sensor 2004 and a controller 2006. Furthermore, similar to the depth sensing module of Figure 2, the depth sensing module of Figure 5 may also, optionally, comprise a detector 2008.

[0082] Therefore, further discussion of the features of the depth sensing module illustrated in the example of Figure 4 which are the same as those features which have already been described in the example of Figure 2 will not be provided again at this stage.

[0083] However, in this specific example of Figure 5, the illuminator 5002 may be connected to, otherwise include, an optical element 5004, which can be used in order to change a visual field of the illuminator 5002. In examples, the optical element 5004 may thus be provided between the illuminator 5002 and the scene which is to be illuminated.

[0084] In examples, the optical element 5004 may include may include one or more refractive or reflective elements (or a combination of both refractive and reflective elements). For example, the optical element 5004 may be a lens system (e.g. a combination of different lenses which can be used, together, to change the visual field of the illuminator 5004). The optical element (or the components thereof) may be connected to an actuating element (not shown in Figure 5 of the present disclosure) which enables the configuration of the optical element (and thus its optical properties) to be changed, in order to change the visual field of the illuminator 5004.

[0085] The type of actuating element used in order to provide actuation of the optical element is not particularly limited in accordance with embodiments of the disclosure. For example, the actuating element may be an alternating current motor, a direct current motor, a linear motor, a stepper motor, an induction motor, a piezoelectric motor, or the like. Indeed, any suitable actuation element can be used in accordance with embodiments of the disclosure, provided that, through its actuation, it is able to provide actuation of the optical element.

[0086] In this example, the control of the optical element is performed by the controller 2006. Therefore, the controller 2006 is configured to perform control to change the illuminated portion of the scene by controlling a configuration of the optical element (which can, in turn, be controlled by controlling actuation of the optical element).

[0087] Consider, for example, a situation where the illuminator is configured, once activated, to emit light of a first wavelength range within a first visual field to illuminate a portion of the scene. The visual field in which the illuminator can emit light may be defined by the region A2 as illustrated in Figure 5 of the present disclosure. In other words, this region A2 as illustrated in Figure 5 forms the portion of the scene which can be illuminated by the illuminator and, thus, the portion of the scene for which depth information (based on a detection of reflected light from the scene) can be generated.However, this region A2 forms only a portion of the total scene (the region S in Figure 5). Therefore, based on the current visual field illuminated by the illuminator, the depth sensing module would not be able to generate depth information for any portion of the scene outside this region A2. However, in accordance with embodiments of the disclosure, the controller 2006 is able to perform control to change the illuminated portion of the scene. In particular, in this example, the controller is able to perform control to change the illuminated portion of the scene by controlling the configuration of the optical element 5004.

[0088] For example, the controller 2006 may control the configuration of the optical element 5004 such that the portion of the scene S illuminated by the illuminator changes from the portion A2 to the portion B2 illustrated in Figure 5 of the present disclosure. In this way, depth information can be generated, by detection of reflected light from the scene, from region B2 - even though this region B2 was originally located outside the portion A2 of the scene illuminated by the illuminator.

[0089] Notably, the total area illuminated by the illuminator will remain constant, if no change in the level of power supplied to the illuminator occurs. Therefore, even though the optical element 5004 changes the visual field illuminated by the illuminator 5004, the area of the visual field A2 and the visual field B2 will be the same (for a constant power level). However, a change in the visual field through control of the optical element 5004 can increase the field of view (and decrease the range) of the illuminator or decrease the field of view (and increase the range) of the illuminator, depending on the specific portion of the scene (e.g. the current region of interest) which should be illuminated.

[0090] In examples, the controller may control the optical element such that the illuminator illuminates different portions of the scene in turn (e.g. through a sweeping pattern). Therefore, even though only two regions of the scene (regions A2 and B2) are shown as illuminated by the illuminator in Figure 5, it will be appreciated that the present disclosure is not particularly limited in this respect. However, the entirety of the scene does not necessarily have to be illuminated by the illuminator: in examples, the controller may perform control to change the illuminated portion of the scene by controlling the optical element 5004 such that a certain target region (e.g. a region of interest) is illuminated. This enables the depth sensing module to acquire depth sensing information of any relevant section of the scene as required. Specific targeting of a region of interest of the scene may be based on the one or more conditions detected by the detector (if present) and will be discussed in more detail later.

[0091] Therefore, it will be appreciated that even though the illuminator 3002 is able to illuminate only a portion of the scene, the depth sensing module can still generate depth information for any relevant section of the scene through control to change the illuminated portion of the scene. Furthermore, it will be appreciated that, in some examples, the control to change the illuminated portion of the scene can be performed by controlling a configuration of an optical element of the depth sensing module.

[0092] In addition, it will be appreciated that, in some examples, control of the configuration of the optical element can be combined with control of the actuation of the illuminator and / or control of the power supplied to the illuminator as required.

[0093] <Plural ity of llluminators>

[0094] Consider, now, the example of Figure 6 of the present disclosure. Figure 6 illustrates an example depth sensing module in accordance with embodiments of the disclosure.The depth sensing module of Figure 6 of the present disclosure is similar to the depth sensing module which has been described with reference to Figure 2 of the present disclosure, in that it comprises a sensor 2004 and a controller 2006. Furthermore, similar to the depth sensing module of Figure 2, the depth sensing module of Figure 5 may also, optionally, comprise a detector 2008. However, compared to the example of Figure 2 of the present disclosure, the depth sensing module of Figure 6 comprises both an illuminator 6002A and an illuminator 6002B. In other words, in depth sensing module of Figure 6, a plurality of illuminators are provided (two illuminators - namely, illuminator 6002A and 6002B in this example).

[0095] In examples, the illuminator 6002A and the illuminator 6002B may be configured in order that, once activated, they illuminate a different visual field. That is, the visual field illuminator 6002A may illuminate may be different from the visual field which the illuminator 6002B may illuminate.

[0096] Accordingly, the controller 2006 may be configured to perform control to change the illuminated portion of the scene by controlling activation of one of the illuminator 6002A and illuminator 6002B. Consider, for example, a situation where the illuminator 6002A is configured, once activated, to emit light of a first wavelength range within a first visual field to illuminate a portion of the scene. The visual field in which the illuminator 6002A can emit light may be defined by the region A3 as illustrated in Figure 6 of the present disclosure. In other words, this region A3 as illustrated in Figure 6 forms the portion of the scene which can be illuminated by the illuminator 6002A and, thus, the portion of the scene for which depth information (based on a detection of reflected light from the scene) can be generated.

[0097] However, this region A3 forms only a portion of the total scene (the region S in Figure 6). Therefore, based on the current visual field illuminated by the illuminator, the depth sensing module would not be able to generate depth information for any portion of the scene outside this region A3. However, in accordance with embodiments of the disclosure, the controller 2006 is able to perform control to change the illuminated portion of the scene. In particular, in this example, the controller is able to perform control to change to change the illuminated portion of the scene by controlling activation of one or the illuminator 6002A or the illuminator 6002B.

[0098] For example, the controller 2006 may control the activation of the illuminators 6002A and 6002B, such that the controller activates illuminator 6002B instead of illuminator 6002A. Accordingly, the portion of the scene S illuminated changes from the portion A3 to the portion B3 illustrated in Figure 6 of the present disclosure (since the illuminator 6002B is configured to illuminate a different visual field compared to the illuminator 6002A). In this way, depth information can be generated, by detection of reflected light from the scene, from region B3 - even though this region B3 was originally located outside the portion A3 of the scene illuminated by the illuminator 6002A.

[0099] In other words, the controller may determine to control activation of an illuminator (e.g. either illuminator 6002A or 6002B) in view of the different visual field illuminated by each of the illuminators in order to cause a desired portion of the scene to be illuminated.

[0100] In this example, the illuminator 6002A has a wider field of view, but a shorter range. On the other hand, the illuminator 6002B has a narrower field of view, but a longer range. The controller may therefore perform control to illuminate the region A3 or the region B3 as required depending on the situation to which embodiments of the disclosure are applied.

[0101] By activating only one of the illuminators 6002A and 6002B at any given time, the total power consumption of the depth sensing module can be maintained at a same level as if only a singleilluminator was present in the depth sensing module. Nevertheless, since the controller can cause activation of one of the illuminator 6002A or the illuminator 6002B as desired, a different portion of the scene can be illuminated as required. This improves the range over which the depth information can be generated, thus enabling improved measurement of depth information of a scene to be made.

[0102] While the example of Figure 6 of the present disclosure has been described with reference to a situation where the depth sensing module comprises two illuminators (i.e. illuminator 6002A and illuminator 6002B) it will be appreciated that the present disclosure is not particularly limited in this regard. That is, the depth sensing module of the present disclosure may, alternatively, comprises a plurality of illuminators including any number of illuminators (including a number much bigger than two). Each of these illuminators may, optionally, have a different visual field. Accordingly, the controller 2006 may be configured to control activation of one or more of the plurality of illuminators which are provided in the depth sensing module in order to illuminate a different portion of the scene with the light of the first wavelength range as required, depending on the situation to which the embodiments of the disclosure are applied. As such, while only two regions A3 and B3 can be illuminated in the example of Figure 6 of the present disclosure, it will be appreciated that the present disclosure is not particularly limited in this regard.

[0103] In some examples, the controller may perform control to control activation of one of the illuminator 6002A and illuminator 6002B to control switching between activation of the illuminator 6002A and illuminator 6002B of the depth sensing module. That is, by switching between the illuminator 6002A and illuminator 60024 (e.g. based on a fixed schedule or the like) depth information may be generated for both the region A3 and the region B3 (with the depth information for region A3 being generated during activation of the illuminator 6002A and depth information for region B3 being generated during activation of the illuminator 6002B). In examples, said switching may be based on a switching schedule (e.g. with activation of the illuminator 6002A and illuminator 6002B being controlled at a fixed interval of time - such as every few seconds). However, in other examples, the controller may be configured to control switching between activation of the illuminator 6002A and the illuminator 6002B in accordance with a detection of one or more conditions by the detector (if provided). In this way, the controller may target a certain region of interest within the scene. Specific targeting of a region of interest of the scene may be based on the one or more conditions detected by the detector (if present) and will be discussed in more detail later.

[0104] Furthermore, in examples, the depth sensing module of Figure 6 (comprising a plurality of illuminators) may be combined with the depth sensing module of Figures 3, 4 or 5 of the present disclosure. That is, the controller 2006 of the depth sensing module may also perform control to change an illuminated portion of the scene by controlling actuation of one or more of the plurality of illuminators, a level of power supplied to one or more of the plurality of illuminators, and / or the configuration of an optical element (or optical elements) connected to or forming part of one or more of the plurality of illuminators of the depth sensing module as required depending on the situation to which the embodiments of the disclosure are applied.

[0105] <Detector>

[0106] Consider, now, Figures 7 and 8 of the present disclosure. Figures 7 and 8 each illustrate an example depth sensing module in accordance with embodiments of the disclosure.

[0107] The example depth sensing modules of Figures 7 and 8 are similar to the example depth sensing module which has been described with reference to Figure 6 of the present disclosure. That is, thedepth sensing module of Figures 7 and 8 each comprises a sensor 2004, an illuminator 6002A, an illuminator 6002B and a controller 2006. However, in addition, the depth sensing modules of Figures 7 and 8 each further comprises a detector 2008.

[0108] The detector 2008 is configured to detect one or more conditions, with the controller being configured to perform control to change the illuminated portion of the scene in accordance with the one or more detected conditions. For example, the controller is configured to perform control to switch between illuminator 6002A and illuminator 6002B based on a detection of the one or more conditions by the detector.

[0109] In the example of Figure 7, the example condition detected by the detector is an input by a user. That is, in the example of Figure 7, a user 7000 provides an input instruction which is detected by the detector 2008, with the input instruction being provided in order to make a selection between the use of illuminator 6002A and illuminator 6002B of the depth sensing module.

[0110] The user input provided by the user 7000 may be provided through a user input device 1006 such as that which has been described with reference to Figure 1 of the present disclosure, for example. For example, the user input may be provided through a user interface, a touch screen display, a voice command, a gesture or the like. Upon receiving the user input, the detector 2008 (which may be implemented as processing circuitry such as the processing circuitry 1002 described with reference to Figure 1 of the disclosure) the may detect a certain condition (e.g. input instruction). As an example, a first type of user input (e.g. a first gesture) may correspond to a condition of an instruction to activate the illuminator 6002A, while a second, different, type of user input (e.g. a second gesture) may correspond to a condition of an instruction to activate the illuminator 6002B. In this way, the user may provide an instruction which causes the controller to change the currently illuminated portion of the scene.

[0111] An example situation whereby the user input may be provided may be a situation where the depth sensing module is implemented as part of a vehicle. The user may then be a driver of the vehicle; the user may determine to change the currently illuminated portion of the scene (and thus the portion of the scene for which depth information is generated) depending on a current driving condition or the like.

[0112] Accordingly, the example of Figure 7 provides an example wherein the one or more conditions include an input by the a user, and wherein the controller is configured to control switching between activation of the illuminator 6002A and the illuminator 6002B in accordance with a detection of the input by the user.

[0113] The detection of user input, by the detector 2008, is merely one example of a condition which can be detected by the detector. In the example of Figure 8 of the present disclosure, the detector 2008 does not receive input from a user. Rather, in the example of Figure 8 of the present disclosure, the detector 2008 receives an input from one or more sensors 2010.

[0114] In this example, the one or more sensors 2010 are part of the depth sensing module. However, in other examples, the one or more sensors 2010 may be external to the depth sensing module. For example, the detector 2008 may be configured to receive a signal from one or more external sensors 2010 via any suitable wired or wireless connection. In examples, the external sensors may be part of an external device in which the depth sensing module is implemented: as an example, when the depth sensing module is used for depth sensing in a vehicle, the external sensors may be part of the vehicle.The type of sensor and the type of information they are configured to sense is not particularly limited in accordance with embodiments of the disclosure. As an example, the sensor may comprise at least one of a pressure sensor, a temperature sensor, a humidity sensor, a light sensor, an accelerometer, a motion sensor, a sound sensor, a smoke sensor, or the like. The condition detected by the detector may depend on the type of sensor 2010. For example, when the sensor incudes a temperature sensor, the condition may include a detection that the temperature measured by the temperature sensor has exceeded a certain value. Alternatively, when the sensor 2010 includes a sound sensor, the condition may include a detection that a sound of a certain type has been measured by the sound sensor, or that a sound of a certain volume has been measured by the sound sensor.

[0115] While the examples of Figures 7 and 8 have been described with reference to a depth sensing module similar to that described with reference to Figure 6 of the present disclosure (i.e. a depth sensing module with a plurality of illuminators) it will be appreciated that the present disclosure is not particularly limited in this regard. A detector 2008 may also be used in accordance with any of the depth sensing modules as described with reference to Figures 2 to 5 of the present disclosure. Based on the detected condition, the controller may then perform control to change the illuminated portion of the scene. Further examples of the type of control which may be performed by the controller upon detection of a certain condition by the detector 2008 (based on the information detected by one or more sensors or sensing devices) will be described in more detail later with reference to Figure 11 of the present disclosure.

[0116] Furthermore, while a number of example conditions detected by the detector have been described with reference to Figures 7 and 8 of the present disclosure, it will be appreciated that the present disclosure is not particularly limited to these specific examples. Further examples of conditions which can be detected by the detector and used, by the controller, for control of the illuminated portion of the scene will be described later with reference to the example of Figure 11 of the present disclosure.

[0117] <Sensor>

[0118] As has been explained with reference to Figure 2 of the present disclosure, the configuration of the sensor 2002 of the depth sensing module is not particularly limited in accordance with embodiments of the disclosure, provided that the sensor is able to detect the light of the first wavelength range reflected form the illuminated portion of the scene.

[0119] However, in some examples, the sensor may, optionally, be further configured to detect light of a second wavelength range from the scene. In some examples, the light of the first wavelength range may be light of a non-visible wavelength range (e.g. infrared light) while the light of the second wavelength range may be light of a visible wavelength range (e.g. visible light, which can be detected by the eye of a person (such as light of the wavelength range 400 nanometres to 700 nanometres)). Indeed, in examples, the sensor may comprise a so-called combined, or stacked, sensor, which is able to detect both light of the first wavelength range and light of the second wavelength range. Consider, now, Figure 9 of the present disclosure. Figure 9 illustrates a sensor in accordance with embodiments of the disclosure. The sensor of Figure 9 is an RGB-D coaxial sensor, where the RBG layer and the depth sensing layer are stacked in different layers of a sensor.

[0120] In the example of Figure 9, the sensor provides a dual sensing layer. In examples, one of the sensing layers may be configured to detect visible light (e.g. R,G, B) while one of the sensing layers may beconfigured to detect light of the non-visible wavelength range (e.g. infrared light). Accordingly, in examples, the sensor may be a single-aperture RGB + SPAD sensor, that provides single-aperture RGB imaging capabilities and SPAD sensing capabilities.

[0121] In the example of Figure 9, a dual band pass filter 9000 (DBPF) is provided. This is arranged above a dual sensing layer of the sensor.

[0122] The dual sensing layer comprises a thin RGB sensitive layer 9002 (photodiode layer) with no or little infrared absorption. This RGB sensitive layer 9002 may, for example, comprise Silicone photodiodes or organic photodiodes. The dual sensing layer may further comprise a visible-cut filter 9004 for residual visible light removal. Then, the dual sensing layer further comprises a Silicon (NIR) or Germanium (SWIR)- based SPAD layer 9006 for time-of-flight measurement and detection. The SPAD layer 9006 may, for example, comprise Germanium on Silicon or Germanium-based Single photon avalanche diodes. A readout layer 9008 in CMOS technology may be arranged below the dual sensing layer. The readout layer 9008 may, for example, be realized as a CMOS readout and processing layer. The connectivity for the SPADs may, for example, be realized as a high density wafer-to-wafer connectivity (e.g. by hybrid bonding).

[0123] The sensor of Figure 9 is an example of a type of sensor which may be configured in order to detect light of both the first and second wavelength range. However, the present disclosure is not particularly limited to this specific example.

[0124] When a sensor is provided in the depth sensing module which is capable of detecting both the light of the first and second wavelength range (for example, non-visible and visible light) the controller may be configured to generate an image of the scene using the detected light of the second wavelength range (with depth information being generated, for the illuminated portion of the scene, based on the detected light of the first wavelength range).

[0125] For example, the depth sensing module may utilise the information from the depth sensing layer (e.g. the SPAD layer in the example of Figure 9) in order to generate the depth information from the scene. On the other hand, at the same time, the depth sensing module may also utilise the information from the visible light layer (e.g. the RGB sensitive layer of the example of Figure 9) in order to generate an image of the scene. The image of the scene may be generated in any format and may be output, for example, for display to a user.

[0126] It will be appreciated that when the second wavelength range is a visible wavelength range, the image generated from the detected light of the second wavelength range is an image generated based on ambient light of the second wavelength range. That is, the illuminator of the sensor module emits only light of the first wavelength range (e.g. infrared light). Light of the second wavelength range may be ambient light (e.g. from an external light source). In some examples, the ambient light may be light from the sun (i.e. daylight). However, the present disclosure is not particularly limited in this regard and, in some examples, a light source for the second wavelength range may, optionally, also be included as part of the sensor module or its implementation (e.g. as part of a system with the sensor module).

[0127] When the light of the second wavelength range is ambient light, it will be appreciated that the image of the scene can be generated for the entire scene which falls within the field of the view of the image sensor. That is, the depth information can only be generated for a region of the scene which falls within a current visual field of the illuminator (since the depth information is generated based on the light reflected from the scene having been emitted by the illuminator). An example of this has been described with reference to Figure 3 of the present disclosure, for example. In contrast, theambient light of the second wavelength is light which illuminates the entire scene: the image which is generated is then generated for the entire scene which falls within the field of view of the sensor. <lmage Analysis>

[0128] As has been explained with reference to Figures 2 to 8 of the present disclosure, the controller of the present disclosure is configured to generate depth information on the basis of the detected light and perform control to change an illuminated portion of the scene. This control, to change the illuminated portion of the scene, can include actuation of the illuminator, control of an optical element connected to the illuminator, switching between a number of different illuminators (each with a different visual field of illumination, for example), a change of power supplied to the illuminator or the like.

[0129] Furthermore, as has been explained with reference to Figure 9, in examples, the sensor module may be configured to generate an image of the scene, based on detected light of a second wavelength range (e.g. a visible wavelength range).

[0130] Accordingly, in examples, the control to change an illuminated portion of the scene may be based on the image of the scene which has been generated. In other words, in examples, the controller may be configured to change an illuminated portion of the scene in accordance with the image of the scene generated using the detected light of the second wavelength range.

[0131] As such, the controller may, for example, use visible light of the scene to determine a region of interest within the scene which should be illuminated by the illuminator such that depth information can be obtained for that region of interest.

[0132] As has been explained, the image of the scene may be generated for the entire scene (that is, the entire scene which falls within the field of view of the sensor) while the depth information is limited to a portion (or region) of the scene which is currently illuminated by the illuminator. This is because the depth information - based, for example, on time-of-flight - is dependent upon the light emitted by the illuminator, while the image which is generated may be based on ambient light of the scene. As such, the controller can use the image which is generated in order to identify a region of interest for which the depth information should be obtained. This can then be used to control the illuminator such that the depth information can be obtained for the region of interest. The region of interest may be identified based on a detection of an object within the generated image. The region of interest may be identified based on a movement of an object within the generated image. The region of interest may be identified based on a type of object within the generated image. The region of interest may be identified based on an image property of the generated image (e.g. a brightness, contrast, colour, intensity or the like of the generated image or a region thereof).

[0133] For example, if a certain type of object (e.g. a person) is detected in the generated image of the second wavelength range, then the image region containing that person may be identified as the region of interest. In such a situation, the controller may then control the illuminator such that the illuminator emits light in this region of interest (e.g. by actuation of the illuminator) such that depth information can be obtained for this region of interest - enabling depth information for the object in the scene (e.g. the person) to be generated.

[0134] In this way, improved measurement of depth information of a scene can be made, since the depth information can be more accurately and reliably targeted for a region of interest within the scene based on the generated image of the scene.Consider, now, Figure 10 of the present disclosure. Figure 10 illustrates an example method in accordance with embodiments of the disclosure.

[0135] The example method is a method of controlling the illumination provided by an illuminator of a depth sensing module in accordance with a captured image (e.g. an image captured by a visible light layer of a sensor (such as the RGB sensing layer of the example sensor described with reference to Figure 9 of the present disclosure)).

[0136] The example method of Figure 10 starts at step S10000. In steps S10000 and S10002, the depth sensing module is configured to begin depth sensing of a scene. This is performed by activating the illuminator of the sensor module, such that it emits light of the first wavelength range within a first visual field in order to illuminate a portion of the scene. At the same time, the depth sensing module begins to generate image data - using the RGB sensing layer in this example.

[0137] Then, in step S10004, image detection is performed. That is, image processing is performed on the image which has been generated in order to detect one or more features of the image. As discussed, this can include, for example, the presence of an object or a type of object within the image data which has been generated.

[0138] In step S10006, a decision is then made, on the basis of the image detection which has been performed, whether the illumination condition of the illuminator of the sensor module should be changed. For example, it is determined whether to control the illuminator in order to change the illuminated portion of the scene.

[0139] If the decision is positive (i.e. if it is determined to change the illuminated portion of the scene) the controller of the sensor module may then, in step S10008, perform control in order to change the illuminated portion of the scene on this basis. For example, if a certain object, or type of object, has been detected in the image data outside the visual field of the illuminator (that is, the portion of the scene currently illuminated by the illuminator) then the controller may control the illuminator such that the light is emitted and illuminates the portion of the scene which includes the detected object (in order that depth information for this object can be generated). In examples, the control may be performed by actuation of the illuminator. However, the present disclosure is not particularly limited in this regard - any suitable control of the illuminator to change the illuminated portion of the scene can be performed.

[0140] The processing then returns to step S10002, where further image data of the scene is obtained. On the other hand, if, at step S10006, it is determined not to perform control to change the illumination condition (e.g. if the detected object falls within the illuminated portion of the scene) then the processing may return directly to step S10002.

[0141] This process may continue until it is determined that image should stop (step S10010). This may be determined, for example, once the necessary depth information has been generated.

[0142] <Example lmplementation>

[0143] An example implementation of a depth sensing module will now be described with reference to Figure 11 of the present disclosure.

[0144] In this example, the depth sensing module of the present disclosure is provided on a vehicle 11000. In examples, the vehicle 11000 may be an autonomous or semi-autonomous vehicle.The use of a depth sensing module of the present disclosure on a vehicle is one example implementation of the depth sensing module. In other examples, the depth sensing module may be implemented in robotic devices, inspection and monitoring devices, image capture devices or the like.

[0145] The vehicle 11000 shown in the example of Figure 11 has a depth sensing module of the present disclosure, which comprises a stacked sensor such as that described in Figure 9 of the present disclosure. That is, the sensor of the depth sensing module has a depth sensing layer (e.g. a SPAD layer) and a visible light image sensing layer (e.g. an RGB layer).

[0146] The field of view VI of the visible light image sensing layer and the current visual field DI of the depth sensing layer is shown in the example of Figure 11. The field of view VI of the visible light image sensing layer extends beyond the current visual field DI of the depth sensing layer (i.e. the currently illuminated portion of the scene). Accordingly, the image for the scene can be generated across the entire field of view of the visible light image sensing layer, while depth information can be generated only within the current visual field of the depth sensing layer.

[0147] In the context of the vehicle (being, for example, an autonomous or semi-autonomous vehicle), the depth information may provide important distance information which cannot be generated from the RGB image data. For example, the depth information can be used in order to measure the distance to an object, such as a nearby vehicle. Based on this measurement of the distance to other vehicles, the vehicle may change its driving condition (e.g. change its speed, change its direction of travel or the like).

[0148] The region for which depth information may be required may change depending on the type of driving being performed (including, for example, the speed of the vehicle). For example, when a parking manoeuvre is being performed, it may be necessary to measure distance to nearby objects across a wide visual field. However, depth information for objects that are at a further distance from the vehicle may not be required (as they are not relevant for the vehicle when performing a parking manoeuvre). On the other hand, when travelling at speed, the depth information may be required for a vehicle in a same lane as the vehicle at an increased distance (e.g. so the vehicle can determine how far away a leading car in the same lane is to that vehicle). However, depth information may not necessarily be required for cars in adjacent lanes (e.g. at a wide visual field). On the other hand, information of cars in adjacent lanes may become relevant if the vehicle is to change between lanes on the highway, for example. Accordingly, the region for which depth information may be required may change depending on the type of driving being performed. More generally, the region for which depth information may be required may change in accordance with one or more detected conditions (with those conditions being detected by the detector 2008).

[0149] Accordingly, in accordance with embodiments of the disclosure, the controller 2006 of the depth sensing module may perform control in order to change the illuminated portion of the scene (and thus change the portion of the scene for which depth information can be generated) based on one or more conditions detected by the detector 2008.

[0150] As an example, when the depth sensing module is implemented as part of a vehicle as illustrated in the example of Figure 11, the depth sensing module may be configured to receive information from an external system (e.g. a navigation system) which can be used in order to detect (by the detector) the location of the vehicle (e.g. whether the vehicle is on a highway or whether the vehicle is in a car park). Then, if the vehicle is on a highway (an example of a condition detected by the detector) the controller may then control the illuminator to use a narrower illumination with a longer range (e.g.the region D2 in the example of Figure 11). Alternatively, when the vehicle is in a car park (an example of a condition detected by the detector), the controller may control the illuminator to use a wider illumination with a shorter range.

[0151] Alternatively, or in addition, when the depth sensing module is implemented as part of a vehicle as illustrated in the example of Figure 11, the depth sensing system may be configured to receive information from the vehicle (e.g. the speed, tachometer information, location information or the like) which can be used (by the detector) in order to determine whether the vehicle is accelerating or decelerating (an example of a condition detected by the detector). When the vehicle is accelerating, it may be determined, by the controller, to use a narrower, longer range, illumination, for example (e.g. the region D2). In contrast, when the vehicle is decelerating (an example of a condition detected by the detector), it may be determined, by the controller, to use a wider, shorter range, illumination, for example.

[0152] Alternatively, or in addition, when the depth sensing module is implemented as part of a vehicle as illustrated in the example of Figure 11, the depth sensing system may be configured to receive information from an instrument within the vehicle (e.g. an indicator) in order to determine (by the detector) whether the vehicle, or a driver or the vehicle, intends to change the direction of travel of the vehicle (an example of a condition detected by the detector). By receiving this information, the controller may then determine the intended direction of travel of the vehicle and may control the illuminator to illuminate the portion of the scene in the intended direction of travel.

[0153] Alternatively, or in addition, when the depth sensing module is implemented as part of a vehicle, a human driver or passenger in the vehicle may provide input (e.g. a through a user interface, voice command, gesture or the like) to direct the visual field of the illuminator to a specific region of interest (e.g. a region containing a nearby vehicle, for example). Human input is thus an example of a condition which can be detected by the detector, which can be used, by the controller, to control the illuminated portion of the scene. This may ensure that the controller can generate depth information for a region of interest. Alternatively, human input can be used as a ground-truth label for human supervision of the depth sensing module, to further improve the accuracy and reliability of the control performed by the depth sensing module.

[0154] Alternatively, or in addition, in examples, the depth sensing system may be configured to receive information concerning the current weather conditions. This may be based, for example, on a weather forecast, location information (e.g. GPS information), sensor information (e.g. fog detectors, humidity detectors or the like). In examples, the detector may then detect a current or predicted weather condition based on the information which has been received. Then, based on the current or predicted weather conditions, the controller may then control the illuminator to change the illuminated portion of the scene. For example, in bad weather conditions (e.g. fog, heavy rain or the like), it may be determined to use a wider angular range visual field for the illuminator, in order to improve the measurement of depth information which can be performed.

[0155] As has been explained, the visible light imaging (e.g. the RGB image data) may be performed based on ambient light within the scene. Therefore, the visible light imaging may be dependent upon a suitable source of ambient light being present: when the intensity of the ambient visible light is low (e.g. if it is getting dark), it may not necessarily be possible to generate a visible light image of the scene. On the other hand, however, the depth information - generated by the depth sensing layer of the depth sensor - is performed based on the light emitted by the illuminator. Therefore, the depth information may be generated independent of the ambient light conditions. Accordingly, alternatively, or in addition, in examples if the detector determines (as an example condition) thatthe intensity of the visible light decreases (e.g. if the intensity of the visible light decreases below a certain threshold value) it may be determined, by the controller, to increase the angular width of the visual field of the illuminator, such that it covers a wide range of the field of view of the visible light image sensor. In this situation, the depth information may be generated in order to supplement / augment / replace the visible light image data such that the vehicle system has knowledge of the location of objects in the scene (even though, for example, it is too dark to detect those objects in the visible light image data).

[0156] Alternatively, or in addition, in examples, the controller may perform control to change the visual field of the illuminator based on a detected object (the detection of an object being an example condition which can be detected by the detector). For example, if a human is detected (e.g. by deep learning-based object detection on the image data which has been generated) then the controller may change the visual field of the illuminator to illuminate the detected human, thus ensuring that distance information can be generated for a region covering the detected human (e.g. such that the proximity of the human can be accurately and reliably detected).

[0157] Alternatively, or in addition, in examples, the controller may be configured to determine a region which should be illuminated (e.g. a region of interest) in accordance with information received from an external sound monitoring device (e.g. a microphone). For example, if a certain sound, or a certain volume of sound, is detected by the detector based on the information from the external sound monitoring device (corresponding, for example, to a loud sound such as an accident, explosion, alarm, or horn of a vehicle) the controller may be configured to determine the region of interest as the region from which the sound originated. That is, the controller may be configured to perform control to change the region illuminated by the illuminator (e.g. by actuation or the like) in order that the region illuminated by the illuminator coincides with the region from which the sound originated. This then enables the depth sensing module to generate depth information for this region from which the sound has been detected, ensuring that improved measurement of depth information of a scene can be made.

[0158] Alternatively, or in addition, in examples, the controller may be configured to determine a region which should be illuminated for depth measurement (e.g. a region of interest) in accordance with a motion which is detected (with the motion being an example of a condition which can be detected by the detector). The motion may be detected, for example, from the image data which has been generated (e.g. the RGB image data). In examples, the motion may be detected by computing optical flow of frames, receiving information from an accelerometer or the like. In examples, when strong motion is detected (e.g. when the motion of an object exceeds a threshold value) the controller may identify the region containing that object as a region of interest, and may control the illuminator accordingly. In examples, a wider visual field may be determined as the most appropriate illumination from the illuminator in the case that strong motion has been detected. In examples, when a certain object is within the visual field of the illuminator at a certain time, but is predicted to move outside the visual field of the illuminator at a next time (e.g. based on a predicted motion of the object (such as an extrapolation of the movement of the object)) the controller may control the illuminator such that the object will remain within the illuminated portion of the scene at the next instance of time (e.g. by using a wider visual field). This ensures that depth information can be generated for a target object (e.g. a nearby vehicle) even when that target object has strong movement.

[0159] Alternatively or in addition, the controller may be configured to control the illuminator in order to change the illuminated portion of the scene based on a current or predicted occlusion between objects (with the current or predicted occlusion being an example of a condition which can bedetected by the detector). A current or predicted occlusion can be determined based on optical flow and depth information for objects within the scene. In this way, the controller can determine a portion of the scene to illuminate to maximise the depth information which can be obtained around the predicted occlusion.

[0160] Of course, it will be appreciated that the decision processes performed by the detector and controller to determine a portion of the scene to be illuminated is not particularly limited to these specific examples. Other processes may be applied by the detector and / or controller as required depending on the situation to which the embodiments of the disclosure are applied. Furthermore, while certain decision processes performed by the detector and controller to determine whether to change the illuminated portion of the scene have been described in the context of a vehicle (such as an autonomous or semi-autonomous vehicle) it will be appreciated that the present disclosure is not particularly limited in this regard. That is, the same or similar processes may also be applied by the detector and controller when implemented in some other type of device or situation (including, for example, as part of a robotic device, an imaging device, an inspection and monitoring device or the like).

[0161] Furthermore, while, in certain examples, switching between a discrete number of illuminated regions (e.g. regions A or B) has been shown, it will be appreciated that the present disclosure is not particularly limited in this regard. In examples, the controller may be configured to perform control to illuminate any given portion of the scene as required (e.g. based on a condition which has been detected by the detector 2008). Accordingly, any region (such as an intermediate region between regions A and B) may also be illuminated under control of the controller as required (in order that depth information can be generated for this intermediate region, for example).

[0162] <Method>

[0163] Consider, now, Figure 12 of the present disclosure. Figure 12 illustrates an example method in accordance with embodiments of the disclosure.

[0164] The example method of Figure 12 may be performed in order to control a depth sensing module for generating depth information for a scene. The method of Figure 12 may, optionally, be used in order to control a depth sensing module as described with reference to any of the embodiments of the disclosure. As an example, the method of Figure 12 may be used in order to control a depth sensing module such as that described with reference to Figure 2 of the present disclosure.

[0165] In particular, the depth sensing module controlled by the method of Figure 12 of the present disclosure may comprise an illuminator configured, once activated, to emit light of a first wavelength range within a first visual field to illuminate a portion of the scene, a sensor configured to detect light of the first wavelength range from the scene and a controller.

[0166] Accordingly, the example method of Figure 12 (which starts at step S12000) may, in step S12002, comprise generating, with the controller of the depth sensing module, depth information on the basis of the detected light.

[0167] In step S12004, comprise performing control, with the controller of the depth sensing module, to change an illuminated portion of the scene by controlling actuation of the illuminator.

[0168] The method of Figure 12 may then proceed to, and end with, step S12006.

[0169] In this way, improved measurement of depth information of a scene can be made.It will be appreciated that the present disclosure is not particularly limited to the example method which is illustrated in Figure 12 of the present disclosure. For example, a number of the steps of the method can be performed in a different order than that illustrated in Figure 12 and / or may be performed in parallel to each other. In some examples, a number of additional method steps (not shown in this example) may be included in the method as required. Therefore, the present disclosure is not particularly limited to the specific illustration of the method shown in the example of Figure 12.

[0170] <Computer Program>

[0171] Furthermore, it will be appreciated that the methods of the present disclosure may be carried out on conventional hardware (such as that described previously herein) suitably adapted as applicable by software instruction or by the inclusion or substitution of dedicated hardware. Thus, the required adaptation to existing parts of a conventional equivalent device may be implemented in the form of a computer program product comprising processor implementable instructions stored on a non-transitory machine-readable medium such as a floppy disk, optical disk, hard disk, PROM, RAM, flash memory or any combination of these or other storage media, or realized in hardware as an ASIC (application specific integrated circuit) or an FPGA (field programmable gate array) or other configurable circuit suitable to use in adapting the conventional equivalent device. Separately, such a computer program may be transmitted via data signals on a network such as an Ethernet, a wireless network, the Internet, or any combination of these or other networks.

[0172] <Clauses>

[0173] 1) A depth sensing module for generating depth information for a scene, the depth sensing module comprising:

[0174] an illuminator configured, once activated, to emit light of a first wavelength range within a first visual field to illuminate a portion of the scene;

[0175] a sensor configured to detect light of the first wavelength range reflected from the scene; and

[0176] a controller configured to generate depth information on the basis of the detected light and perform control to change an illuminated portion of the scene.

[0177] 2) The depth sensing module according to clause 1, wherein i) the controller is configured to perform control to change the illuminated portion of the scene by controlling actuation of the illuminator or ii) the controller is configured to perform control to change the illuminated portion of the scene by controlling a level of power supplied to the illuminator.

[0178] 3) The depth sensing module according to clause 1 or 2, wherein the depth sensing module further comprises an optical element positioned between the illuminator and the scene, and wherein the controller is configured to perform control to change the illuminated portion of the scene by controlling a configuration of the optical element.

[0179] 4) The depth sensing module according to any proceeding clause, wherein the depth sensing module further comprises a second illuminator configured to emit light of the first wavelength range within a second visual field to illuminate a portion of the scene, the second visual field being different than the first visual field, wherein the controller is configured to perform control to change the illuminated portion of the scene by controlling activation of one of the illuminator or the second illuminator.5) The depth sensing module according to clause 4, wherein controlling activation of one of the illuminator or the second illuminator comprises controlling switching between activation of the illuminator or the second illuminator.

[0180] 6) The depth sensing module according to clause 5, wherein the controller is configured to control switching between activation of the illuminator or the second illuminator in accordance with a switching schedule.

[0181] 7) The depth sensing module according to clause 5, wherein depth sensing module further comprises a detector configured to detect one or more conditions, and wherein the controller is configured to control switching between activation of the illuminator and the second illuminator in accordance with a detection of the one or more conditions.

[0182] 8) The depth sensing module according to clause 7 , wherein the one or more conditions include an input by a user, and wherein the controller is configured to control switching between activation of the illuminator and the second illuminator in accordance with a detection of the input by the user.

[0183] 9) The depth sensing module according to clause 7, wherein the detector is configured to receive input from a sensor and wherein the one or more conditions include: a motion, a presence of a type of object, a location, a speed, an acceleration, a sound, a weather condition, or a light intensity.

[0184] 10) The depth sensing module according to any proceeding clause, wherein the sensor is further configured to detect light of a second wavelength range from the scene.

[0185] 11) The depth sensing module according to clause 10, wherein light of the first wavelength range is light of a non-visible wavelength range and the light of the second wavelength range is light of a visible wavelength range.

[0186] 12) The depth sensing module according to clause 11, wherein the controller is configured to generate an image of the scene using the detected light of the second wavelength range.

[0187] 13) The depth sensing module according to clause 12, wherein the controller is configured to change an illuminated portion of the scene in accordance with the image of the scene generated using the detected light of the second wavelength range.

[0188] 14) A method of controlling a depth sensing module for generating depth information for a scene, the depth sensing module comprising:

[0189] an illuminator configured, once activated, to emit light of a first wavelength range within a first visual field to illuminate a portion of the scene; and

[0190] a sensor configured to detect light of the first wavelength range reflected from the scene; the method comprising generating, with a controller of the depth sensing module, depth information on the basis of the detected light and performing control, with the controller of the depth sensing module, to change an illuminated portion of the scene.

[0191] 15) A computer program which, when implemented by a controller of a depth sensing module comprising an illuminator configured, once activated, to emit light of a first wavelength range within a first visual field to illuminate a portion of the scene and a sensor configured to detect light of the first wavelength range reflected from the scene, causes the depth sensing module to perform a method of:generating, with a controller of the depth sensing module, depth information on the basis of the detected light and performing control, with the controller of the depth sensing module, to change an illuminated portion of the scene.

[0192] In so far as embodiments of the disclosure have been described as being implemented, at least in part, by software-controlled data processing apparatus, it will be appreciated that a non-transitory machine-readable medium carrying such software, such as an optical disk, a magnetic disk, semiconductor memory or the like, is also considered to represent an embodiment of the present disclosure.

[0193] It will be appreciated that the above description for clarity has described embodiments with reference to different functional units, circuitry and / or processors. However, it will be apparent that any suitable distribution of functionality between different functional units, circuitry and / or processors may be used without detracting from the embodiments.

[0194] Described embodiments may be implemented in any suitable form including hardware, software, firmware or any combination of these. Described embodiments may optionally be implemented at least partly as computer software running on one or more data processors and / or digital signal processors. The elements and components of any embodiment may be physically, functionally and logically implemented in any suitable way. Indeed the functionality may be implemented in a single unit, in a plurality of units or as part of other functional units. As such, the disclosed embodiments may be implemented in a single unit or may be physically and functionally distributed between different units, circuitry and / or processors.

[0195] Although the present disclosure has been described in connection with some embodiments, it is not intended to be limited to the specific form set forth herein. Additionally, although a feature may appear to be described in connection with particular embodiments, one skilled in the art would recognize that various features of the described embodiments may be combined in any manner suitable to implement the technique.

Claims

CLAIMS1. A depth sensing module for generating depth information for a scene, the depth sensing module comprising:an illuminator configured, once activated, to emit light of a first wavelength range within a first visual field to illuminate a portion of the scene;a sensor configured to detect light of the first wavelength range reflected from the scene; anda controller configured to generate depth information on the basis of the detected light and perform control to change an illuminated portion of the scene.

2. The depth sensing module according to claim 1, wherein the controller is configured to perform control to change the illuminated portion of the scene by controlling actuation of the illuminator or ii) the controller is configured to perform control to change the illuminated portion of the scene by controlling a level of power supplied to the illuminator.

3. The depth sensing module according to claim 1, wherein the depth sensing module further comprises an optical element positioned between the illuminator and the scene, and wherein the controller is configured to perform control to change the illuminated portion of the scene by controlling a configuration of the optical element.

4. The depth sensing module according to claim 1, wherein the depth sensing module further comprises a second illuminator configured to emit light of the first wavelength range within a second visual field to illuminate a portion of the scene, the second visual field being different than the first visual field, wherein the controller is configured to perform control to change the illuminated portion of the scene by controlling activation of one of the illuminator or the second illuminator.

5. The depth sensing module according to claim 4, wherein controlling activation of one of the illuminator or the second illuminator comprises controlling switching between activation of the illuminator or the second illuminator.

6. The depth sensing module according to claim 5, wherein the controller is configured to control switching between activation of the illuminator or the second illuminator in accordance with a switching schedule.

7. The depth sensing module according to claim 5, wherein depth sensing module further comprises a detector configured to detect one or more conditions, and wherein the controller is configured to control switching between activation of the illuminator and the second illuminator in accordance with a detection of the one or more conditions.

8. The depth sensing module according to claim 7, wherein the one or more conditions include an input by a user, and wherein the controller is configured to control switching between activation of the illuminator and the second illuminator in accordance with a detection of the input by the user.

9. The depth sensing module according to claim 7, wherein the detector is configured to receive input from a sensor and wherein the one or more conditions include: a motion, a presence of a type of object, a location, a speed, an acceleration, a sound, a weather condition, or a light intensity.

10. The depth sensing module according to claim 1, wherein the sensor is further configured to detect light of a second wavelength range from the scene.

11. The depth sensing module according to claim 10, wherein light of the first wavelength range is light of a non-visible wavelength range and the light of the second wavelength range is light of a visible wavelength range.

12. The depth sensing module according to claim 11, wherein the controller is configured to generate an image of the scene using the detected light of the second wavelength range.

13. The depth sensing module according to claim 12, wherein the controller is configured to change an illuminated portion of the scene in accordance with the image of the scene generated using the detected light of the second wavelength range.

14. A method of controlling a depth sensing module for generating depth information for a scene, the depth sensing module comprising:an illuminator configured, once activated, to emit light of a first wavelength range within a first visual field to illuminate a portion of the scene; anda sensor configured to detect light of the first wavelength range reflected from the scene; the method comprising generating, with a controller of the depth sensing module, depth information on the basis of the detected light and performing control, with the controller of the depth sensing module, to change an illuminated portion of the scene.

15. A computer program which, when implemented by a controller of a depth sensing module comprising an illuminator configured, once activated, to emit light of a first wavelength range within a first visual field to illuminate a portion of the scene and a sensor configured to detect light of the first wavelength range reflected from the scene, causes the depth sensing module to perform a method of:generating, with a controller of the depth sensing module, depth information on the basis of the detected light and performing control, with the controller of the depth sensing module, to change an illuminated portion of the scene.

Images