An apparatus, method and computer program for determining location data for an object

Abstract

In accordance with embodiments of the disclosure, an apparatus for determining location data for an object is provided, the apparatus comprising circuitry configured to: acquire, from depth detection circuitry, depth data of a field of view of the depth detection circuitry; identify from the depth data of the field of view, by a trained model, first depth data representing depth data of a reflection of an object and second depth data representing depth data of a reflective surface, the reflective surface being the surface from which the reflection of the object is received; and determine location data for the object for which the reflection of the object is received based on the first depth data in relation to the second depth data, wherein the location data being indicative of a location outside a field of view. A corresponding method and computer program are also provided.

Description

[0001] AN APPARATUS, METHOD AND COMPUTER PROGRAM FOR DETERMINING LOCATION DATA FOR AN OBJECT BACKGROUND

[0002] Field of the Disclosure

[0003] The present disclosure relates to an apparatus, method and computer program for determining location data for an object.

[0004] Description of the Related Art

[0005] The "background" description provided herein is for the purpose of generally presenting the context of the disclosure. Work of the presently named inventors, to the extent it is described in the background section, as well as aspects of the description which may not otherwise qualify as prior art at the time of filing, are neither expressly or impliedly admitted as prior art against the present invention.

[0006] Image capture devices are used in a wide range of technology to acquire images of a scene. For example, image capture devices can be used in portable electronic devices, computing devices, robotic devices, vehicles and the like. Image capture devices can be used to obtain an image using visible wavelengths of light. Image capture devices can also be used in order to obtain depth information (e.g. based on direct time-of-flight). Depth information can be used in order to improve three-dimensional understanding of an environment. For example, depth information can be used in simultaneous localization and mapping (SLAM) to map an unknown environment or can be used in a vehicle navigation system (e.g. to assist in controlling an autonomous, or semi-autonomous vehicle). However, despite finding applications in a wide range of technology, image capture devices can struggle to perform with a desired degree of accuracy when reflections are present within the field of view of the image capture device. These reflections can interfere with image capture performed by the image capture devices. For example, if glass (or another reflective - or partially reflective -surface) is present, depth from the glass or depth from reflected objects can interfere with obtaining correct depth information and focus.

[0007] As such, an aim of the present disclosure is to improve the performance of an image capture device when a reflective surface is included within the field of view of the image capture device.

[0008] SUMMARY

[0009] A brief summary about the present disclosure is provided hereinafter to provide a basic understanding related to certain aspects of the present disclosure.

[0010] Embodiments of the present disclosure are defined by the independent claims. Further aspects of the disclosure are defined by the dependent claims.

[0011] In accordance with embodiments of the disclosure, improved performance of an image capture device when a reflective surface is included within the field of view of the image capture device can be achieved.

[0012] The present disclosure is not particularly limited to these advantageous technical effects. Other technical effects will become apparent for the skilled person when reading the disclosure.

[0013] BRIEF DESCRIPTION OF THE DRAWINGSA more complete appreciation of the disclosure and many of the attendant advantages thereof will be readily obtained as the same becomes better understood by reference to the following detailed description when considered in connection with the accompanying drawings, wherein:

[0014] Figure 1 illustrates an example information processing device in accordance with embodiments of the disclosure;

[0015] Figures 2A and 2B illustrate an example situation;

[0016] Figure 3 illustrates a first example apparatus in accordance with embodiments of the disclosure; Figure 4 illustrates an example point cloud in accordance with embodiments of the disclosure; Figure 5 illustrates an example histogram of a point cloud in accordance with embodiments of the disclosure;

[0017] Figure 6 illustrates an example of a subsurface scattering model in accordance with embodiments of the disclosure;

[0018] Figures 7A to 7C illustrate an example of correcting point cloud geometry in accordance with embodiments of the disclosure;

[0019] Figure 8 illustrates an example situation in accordance with embodiments of the disclosure;

[0020] Figures 9 and 10 illustrate an example of point cloud filtering in accordance with embodiments of the disclosure;

[0021] Figure 11 illustrates an first example method in accordance with embodiments of the disclosure; Figure 12 illustrates a second example apparatus in accordance with embodiments of the disclosure; Figure 13 illustrates an example process flow in accordance with embodiments of the disclosure; Figures 14 and 15 illustrate example point cloud filtering in accordance with embodiments of the disclosure;

[0022] Figure 16 illustrates a second example method in accordance with embodiments of the disclosure; Figure 17 illustrates a third example apparatus in accordance with embodiments of the disclosure; Figure 18 illustrates an example situation in accordance with embodiments of the disclosure;

[0023] Figure 19 illustrates a third example method in accordance with embodiments of the disclosure.

[0024] DESCRIPTION OF THE EMBODIMENTS

[0025] Referring to Figure 1, an example apparatus 1000 (an example of an information processing device) according to embodiments of the disclosure is shown. Typical ly, an apparatus 1000 according to embodiments of the disclosure is a computer device such as a personal computer, an entertainment system or videogame console, or a terminal connected to a server. Indeed, in embodiments, the apparatus may also be a server. The apparatus 1000 is controlled using a microprocessor or other processing circuitry 1002. In some examples, the apparatus 1000 may be a portable computing device such as a mobile phone, laptop computer or tablet -computing device.

[0026] The processing circuitry 1002 may be a microprocessor carrying out computer instructions or may be an Application Specific Integrated Circuit. The computer instructions are stored on storage medium 1004 which maybe a magnetically readable medium, optically readable medium or solid state typecircuitry. The storage medium 1004 may be integrated into the apparatus 1000 or may be separate to the apparatus 1000 and connected thereto using either a wired or wireless connection. The computer instructions may be embodied as computer software that contains computer readable code which, when loaded onto the processor circuitry 1002, configures the processor circuitry 1002 to perform a method according to embodiments of the disclosure.

[0027] Additionally, an optional user input device 1006 is shown connected to the processing circuitry 1002. The user input device 1006 may be a touch screen or may be a mouse or stylist type input device. The user input device 1006 may also be a keyboard, controller, or any combination of these devices. In some examples, the user input device 1006 may be a microphone or other device. The user to may then provide input via sounds or speech.

[0028] A network connection 1008 may optionally be coupled to the processor circuitry 1002. The network connection 1008 may be a connection to a Local Area Network or a Wide Area Network such as the Internet or a Virtual Private Network or the like. The network connection 1008 may be connected to a server allowing the processor circuitry 1002 to communicate with another apparatus in order to obtain or provide relevant data. The network connection 1002 may be behind a firewall or some other form of network security.

[0029] Additionally, shown coupled to the processing circuitry 1002, is a display device 1010. The display device 1010, although shown integrated into the apparatus 1000, may additionally be separate to the apparatus 1000 and may be a monitor or some kind of device allowing the user to visualize the operation of the system (e.g. a display screen or a head mounted display). In addition, the display device 1010 may be a printer, projector or some other device allowing relevant information generated by the apparatus 1000 to be viewed by the user or by a third party.

[0030] As explained in the Background, image capture devices are used in a wide range of technology to acquire images of a scene. For example, image capture devices can be used in portable electronic devices, computing devices, robotic devices, vehicles and the like. Image capture devices can be used to obtain an image using visible wavelengths of light. Image capture devices can also be used in order to obtain depth information (e.g. based on direct time-of-flight). Depth information can be used in order to improve three-dimensional understanding of an environment. For example, depth information can be used in simultaneous localization and mapping (SLAM) to map an unknown environment or can be used in a vehicle navigation system (e.g. to assist in controlling an autonomous, or semi-autonomous vehicle).

[0031] In addition, depth sensing systems can be used in a robotic device (such as a robotic device configured for providing local delivery, a robot device for bringing food / drink in hospitality environments, or a robotic device operating in a factory environment, for example), cleaning systems (such as a vacuum cleaner or a floor cleaner, for example), robotic devices for agriculture and horticulture, robotic devices for harvesting, or robotic devices for providing assistance and services (e.g. in home care and healthcare), for example.

[0032] However, despite finding applications in a wide range of technology, image capture devices can struggle to perform with a desired degree of accuracy when reflections are present within the field of view of the image capture device. These reflections can interfere with image capture performed by the image capture devices. For example, if glass (or another reflective - or partially reflective -surface) is present, depth from the glass or depth from reflected objects can interfere with obtaining correct depth information and focus.Consider, now, Figures 2A and 2B of the present disclosure. Figures 2A and 2B illustrate an example situation. In this example situation, a reflective surface falls within the field of view of an image capture device.

[0033] That is, the example situation of Figures 2A and 2B illustrate a problem that can arise when a reflective surface is present within the field of view of an image capture device.

[0034] As shown in Figure 2A, in this example situation, a camera (a type of image capture device) is being used in order to capture an image of a target object. A polycarbonate plate (of 2cm thickness in this example) is situated between the target and the camera. The polycarbonate plate is an example of a reflective surface. That is, the polycarbonate plate is at least partially transparent, such that light from the target can reach the camera. However, at the same time, the surface of the polycarbonate plate is at least partially reflective, such that light of a portion of the light hitting the surface of the polycarbonate plate is reflected (and thus does not pass through the polycarbonate plate).

[0035] In this example, a second object (the reflected target) is present on the same side of the polycarbonate plate as the camera. A portion of the light from the reflected target will pass through the polycarbonate plate. However, a portion of the light from the reflected target will be reflected by the surface of the polycarbonate plate, such that a reflection of the reflected target is visible to the camera on the surface of the polycarbonate plate. This is shown in Figure 2A - where light from both the target and the reflected target reach the camera.

[0036] In this example, the target and the reflected target are both illuminated by lights. That is, the reflected target is illuminated by Light 1 (a front light, positioned on the same side of the polycarbonate plate as the camera). The target is illuminated by Light 2 (a back light, positioned on the same side of the polycarbonate plate as the target). However, while this set up is used in order to show how the target and reflected target can be illuminated, it will be appreciated that the present disclosure is not particularly limited in this regard. In examples, the target and reflected target may be illuminated by ambient illumination, for example.

[0037] It will be appreciated that, in this example, the camera is trying to capture an image of the target through the polycarbonate plate. However, since the surface of the polycarbonate plate is partially reflective, an image of the reflected target (i.e. the reflection of the reflected target) is also seen by the camera when trying to capture an image of the target. This can make it very difficult for the image capture device to capture an image of the target. In other words, the reflection of the reflected target can interference with the image capture device as it is used to capture an image of the target.

[0038] An example of this is shown in Figure 2B. Figure 2B shows an example lens position of the camera, as it attempts to focus on the target (in order to capture an image of the target). In this example, the camera is being used in an autofocus mode, whereby the camera will automatically determine a lens position to control the focus of the camera such that it focuses on the target. Here, the autofocus mode may be a phase detection auto focus (PDAF). It will be appreciated that PDAF is a known autofocusing method that can detect where light rays ravel and meet when entering a camera (and can thus be used in order to focus the camera on the target). For a PDAF image sensor, a number of pixels of the image sensor are used for phase detection; these pixels may, for example, comprise masked pixels, which can be used in order to provide a "left" and "right" image, which can then be compared for phase detection.

[0039] The ideal lens position for the camera to focus on the target is shown in the example of Figure 2B as lens position 185. Alternatively, if it was desired that the camera focused on the reflection of thereflected target, then the ideal lens position would be 164. However, the PDAF, when trying to focus on the target, determines a lens position of 175 as the appropriate lens position to focus on the target. This is due to the interference from the reflection of the reflected target, which makes it difficult for the PDAF to determine the correct focus position for the target.

[0040] As such, when an autofocus mechanism such as PDAF is used in the presence of a reflective surface, it is very difficult for the autofocus mechanism to achieve a correct focus on the target (meaning that the intended target may be out-of-focus in the captured image). Accordingly, the presence of a reflective surface within the field of view of an image capture device can cause the image capture device to struggle to perform with a desired degree of accuracy and reliability. Furthermore, the reflective surface does not only interfere with autofocus mechanisms such as PDAF, but can - as previously mentioned - also cause problems concerning depth measurements (particularly in the context of depth sensing systems such as SLAM and autonomous navigation).

[0041] While the example of Figure 2 has been described with reference to a situation where a polycarbonate plate has been provided as a reflective surface, it will be appreciated that the present disclosure is not particularly limited in this regard. For example, interference from a reflection may occur when an image capture device is being used to capture an image of a target object through a window. Alternatively, for example, interference form a reflection may occur when a reflective surface (such as a mirror) is present within the field of view of the image capture device.

[0042] Accordingly, while the problems related to interference arising from a reflective surface have been illustrated with reference to the example situation of Figure 2, it will be appreciated that these problems may occur when an image capture device is used in a wide range of different situations. As such, even though certain embodiments of the disclosure will be described with reference to this example situation, it will be appreciated that the present disclosure is not particularly limited in this regard, and more generally embodiments of the disclosure may be applied to any environment in which objects and reflections of objects may be present.

[0043] Nevertheless, based on the example situation of Figure 2, it will be appreciated that when a reflective surface is present within the field of view of an image capture device, this can cause the image capture device to struggle to perform with a desired degree of accuracy and reliability.

[0044] As such, an apparatus, method and computer program are provided in accordance with embodiments of the disclosure.

[0045] <First Example Apparatus>

[0046] Consider, now, the example of Figure 3 of the present disclosure. Figure 3 illustrates an example apparatus 3000 in accordance with embodiments of the disclosure. The apparatus 3000 is an apparatus for determining location information of an object.

[0047] The Apparatus of Figure 3 comprises circuitry 3002.

[0048] The circuitry 3002 of apparatus 3000 is configured to acquire, from depth detection circuitry, depth data of a field of view of the depth detection circuitry.

[0049] The circuitry 3002 of apparatus 3000 is further configured to identify from the depth data of the field of view, by a trained model, first depth data representing depth data of a reflection of an object and second depth data representing depth data of a reflective surface, the reflective surface being the surface from which the reflection of the object is received.Finally, the circuitry 3002 of apparatus 3000 is configured to determine location data for the object for which the reflection of the object is received based on the first depth data in relation to the second depth data, wherein the location data being indicative of a location outside a field of view. In this way, apparatus 3000 is able to perform improved recovery of location data for an object in the presence of a reflective surface. In particular, apparatus 3000 is able to recover location information for an object - based on its reflection - even when that object is located outside of the field of view. Thus, improved performance of an image capture device when a reflective surface is included within the field of view of the image capture device can be achieved.

[0050] A number of additional components may, optionally, be provided as part of apparatus 3000 in addition to the circuitry 3002. That is, as an example, the apparatus 3000 may optionally further comprise a depth sensor 3004 (as an example of depth detection circuitry). That is, the depth sensor 3004 from which the depth data is acquired (or, more generally, the depth detection circuitry) may -in some examples - be part of the apparatus 3000 itself. However, the present disclosure is not particularly limited in this regard. In other examples, the depth sensor 3004 may be external to, and separate from, the apparatus 3000. In this situation, the apparatus 3000 may be communicatively coupled to the depth sensor 3004 and may thus acquire the depth data from the depth sensor 3004 on this basis.

[0051] In examples, the depth sensor 3004, may be a direct time-of-flight based sensor. For example, the depth sensor 3004 may be configured to generate depth data of a field of view (being a field of view covered by the depth sensor 3004) through detection of a first wavelength range of light reflected from objects within the field of view, with the detection performed by the sensor being used in order to generate depth information for the field of view.

[0052] Indeed, in a depth sensing system which utilizes time-of-flight information (such as a LiDAR system), depth information is obtained by measuring the time taken for reflected light emitted an illuminator to return to the sensor. In examples, an illuminator may rapidly pulse to produce the illumination which can then be used in order to "scan" the field of view (and thus produce depth information). In this way, a depth sensor (such as a LiDAR system) may produce illumination (such as laser illumination) which can be used in order to scan the field of view and which can be used, when reflected off objects in the field of view, to determine depth information concerning those objects within the field of view.

[0053] The type of depth sensor 3004 used to detect the depth information, is not particularly limited in the present disclosure, and may include single photon avalanche diode detector based Light Detection and Ranging (LiDAR) systems and direct Time of Flight based sensors, for example.

[0054] That is, in examples, the pixels of the depth sensor 3004 may comprise single photon detectors (i.e. pixels which can detect arrival of a single photon). In examples, the pixels of the depth sensor 3004 may include single photon avalanche diode detectors (SPADs). A SPAD comprises a photodiode with a reverse bias voltage higher than the breakdown voltage of the photodiode. A single photon incident on the SPAD can trigger a self-sustaining avalanche. Therefore, the SPAD is very sensitive with the ability to detect a single photon.

[0055] However, it will be appreciated that the present disclosure is not particularly limited in this regard. Any other type of pixel may be used in the depth sensor 3004 as required.

[0056] In examples, the circuitry 3002 may be implemented as processing circuitry such as the processing circuitry 1002 as described with reference to Figure 1 of the present disclosure. While, in theexample of Figure 3, a single circuitry 3002 is shown, it will be appreciated that the present disclosure is not particularly limited in this regard. In examples, different functions provided by the circuitry 3002 may be distributed amongst one or more separate circuitry or processing units (e.g. acquiring unit 3002A, identifying unit 3002B and determining unit 3002C). Therefore, it will be appreciated that the present disclosure is not particularly limited to the configuration illustrated in the example of Figure 3.

[0057] Further details of the first example apparatus 3000 will now be described.

[0058] <Point Cloud>

[0059] As has been described with reference to Figure 3 of the present disclosure, the circuitry 3002 of apparatus 3000 is configured to acquire, from depth detection circuitry (such as a depth sensor), depth data of a field of view of the depth detection circuitry.

[0060] The depth data is information which indicates the distance of an object from the depth sensor. As has been described, the depth sensor may be a direct time-of-flight based sensor. In this situation, the depth information may be determined based on the time it takes for light emitted by an illuminator to return to the depth sensor having been reflected from objects within the field of view of the depth sensor. In some examples, the depth sensor may move, over time, in order to perform a scan of a scene (i.e. such that the field of view of the camera is changed to cover a portion of the scene which was previously outside the field of view of the depth sensor). However, in other examples, the depth sensor may remain in a fixed location (such that the field of view of the depth sensor covers a fixed portion of a scene).

[0061] Consider, now, Figure 4 of the present disclosure. Figure 4 illustrates an example point cloud in accordance with embodiments of the disclosure.

[0062] That is, Figure 4 of the present disclosure shows the depth information which may be acquired from a depth sensor located at the position of the camera in the example situation of Figure 2 of the present disclosure. In this example, the depth information forms a point cloud, where each point of the point cloud represents a different distance measurement. Taken as a whole, the point cloud provides understanding of the distance of objects within the field of view of the depth sensor from the depth sensor.

[0063] A first cluster of points are shown in proximity to the depth sensor- these points relate to the transparent plate (i.e. the polycarbonate plate shown in the example of Figure 2). In this example, the depth sensor is a direct time-of-flight sensor. Accordingly, these points, related to the transparent plate, are detected when light of a first wavelength range emitted from an illuminator reflects from the transparent plate and returns to the depth sensor (with the distance information being determined based on the time-of-flight). A second cluster of points is also shown at a greater distance from the depth sensor than the transparent plate; these points are the points related to the target. That is, as shown in the example of Figure 2, the target is located behind the transparent plate at a total distance of 1.6m from the depth sensor (with the transparent polycarbonate plate being located 0.3m from the depth sensor). Since the target is located further from the depth sensor, it will take longer for light emitter from an illuminator to reach the target and return to the depth sensor (leading to an increased time-of-flight and thus an increased distance measurement to the target).

[0064] Between the depth sensor and the target, a third cluster of points is shown in the example of Figure 4. These points relate to the reflected target. That is, light from the illuminator reflects from thesurface of the transparent plate, reflects off the target, then reflects from the surface of the transparent plate and returns to the depth sensor (where a time-of-flight measurement takes place). As such, the depth sensor determines depth information for the reflected target. The reflective surface is located at a distance of 0.3m from the depth sensor, and the reflected target is located at a distance of 0.4m from the reflective surface, such that the reflected target has a total distance of 0.7m from the depth sensor. Accordingly, in the distance information received from the depth sensor, the reflected target (located at a distance of 0.7m) is located at a distance between the reflected plate (having a distance of 0.3m from the depth sensor) and the target (having a distance of 1.6m from the depth sensor).

[0065] Notably, while the depth sensor provides depth information of the entire field of view (including the target, reflected surface and reflected target), this depth information is based only on the distance from the depth sensor. Therefore, the distance information of the reflected target does not show the reflected target at its actual real life location (on the same side of the reflecting surface as the depth sensor) but rather shows the reflected target at its distance from the depth sensor (with this distance between a distance between the distance to the transparent plate (reflective surface) and the target).

[0066] Consider, now, Figure 5 of the present disclosure. Figure 5 illustrates an example histogram of a point cloud in accordance with embodiments of the disclosure.

[0067] More specifically, Figure 5 shows an example of the depth information that may be acquired by the circuitry 3002 of the apparatus 3000 for a single pixel of the depth sensor. In contrast to the example of Figure 4 (which showed a distance projection of the information), Figure 5 shows this same depth information plotted as a histogram (for a single pixel).

[0068] The horizontal axis of the histogram of Figure 5 shows the distance from the depth sensor (i.e. the distance information) while the vertical axis of the histogram shows the number of measurements which were received at a given distance from that pixel of the .

[0069] The full point cloud is formed by putting together the information from all the pixels of the depth sensor (or after the full field of view is scanned (e.g. if used as part of a scanning system (such as a scanning LiDAR)).

[0070] As can be seen in the example of Figure 5, there are three distinct peaks present in the distance information. The first of these peaks corresponds to the transparent plate, the second of these peaks corresponds to the reflected object and the third of these peaks (at the greatest distance from the depth sensor) corresponds to the target object.

[0071] In this example set-up (based on the example situation of Figure 2) it is known which peak in the depth information corresponds to which of the objects present in the scene (i.e. it is known which peak corresponds to the transparent plate, which peak corresponds to the reflected object and which peak corresponds to the target). However, more generally, it may not be known which of the peaks corresponds to which of the objects. Indeed, it may not necessarily be known whether or not a reflective surface is even present within the field of view of the depth sensor. Accordingly, in a typical situation for which an image capture device may be used (e.g. as part of a SLAM system or as part of a navigation system), the apparatus may receive distance information from the depth sensor which forms, for example, a multi-peak point cloud. However, the apparatus may not know which clusters (or peaks) in the point cloud correspond to which objects.Accordingly, before the depth information from the depth sensor can be used in order to provide improved image capture performance of an image capture device in the presence of a reflective surface, it is first necessary for the circuitry of apparatus 3000 to perform identification processing in order to identify data representing a reflected object and / or a reflective surface within the field of view of the depth sensor.

[0072] <Subsurface Scattering>

[0073] As has been explained, the circuitry 3002 of apparatus 3000 is configured to identify from the depth data of the field of view, by a trained model, first depth data representing depth data of a reflection of an object and second depth data representing depth data of a reflective surface, the reflective surface being the surface from which the reflection of the object is received.

[0074] As has been described with reference to Figures 4 and 5 of the present disclosure, this identification process is required in order that the apparatus can determine which portions of the depth information received from the depth sensor relate to reflections from a reflected object (from a reflective surface). In other words, it is necessary to determine which portion of the depth information relates to an object (such as the target object) and which portion of the depth information relates to a reflection of an object (such as a reflection of the reflected object).

[0075] The inventors have realised that it is possible to use changes in the way light scatters from the surface of an object to distinguish between distance measurements to an object and distance measurements to a reflection of an object.

[0076] Subsurface scattering describes the way in which light penetrates the surface of an object and is scattered through interactions with the material of the object. Owing to subsurface scattering, a portion of light reflected for an object may reflect off the object at a different angle than it would have had it been reflected directly from the surface of that object. Furthermore, a portion of that light reflects at the same angle but with a slight delay due to the scattering. That is, light which reflects from the surface of the object will have a slight delay compared to light which penetrates the surface of the object and is scattered through interactions with the material of the object. The nature of the subsurface scattering will depend on the material properties of the object from which the light is reflecting; for example, the subsurface scattering may increase as the translucency of the object increases.

[0077] Accordingly, in the context of a distance measurement (such as a direct time-of-flight distance measurement) the subsurface scattering will change how the distance measurements appear for different objects compared to the reflection of objects. Say, for example, that an object (such as the target) is formed from a first material and the reflected object is made from this same first material, the subsurface scattering of light directly from the object and the reflected object would be the same. However, the depth sensor does not receive the light directly from the reflected object. Rather, the depth sensor performs a distance measurement based on the reflected light from the reflected object, reflected from the reflective surface. This reflection of the light from the reflected object from the reflective surface changes the subsurface scattering (since subsurface scattering occurs also as the light is reflected from the reflective surface) which thus causes a perceptible change in the depth measurements that are performed for the target and the depth measurements which are performed for the reflected target (based upon its reflection).

[0078] This difference can be used by circuitry 3002 of apparatus 3000 in order to perform an identification of first depth data representing depth data of a reflection of an object and second depth datarepresenting depth data of a reflective surface, the reflective surface being the surface from which the reflection of the object is received. In other words, this difference can be used by circuitry 3002 to identify any reflections and reflective surfaces which are present within the field of view of the depth sensor (such that these can be distinguished from real objects in the field of view).

[0079] In examples, a subsurface scattering model can be used for the different peaks in the measured histogram of depth data received from the depth sensor. That is, in examples, the subsurface scattering model can be fit to each measured peak. The parameters of the model (i.e. those parameters which are required in order to achieve a fit to the peak) can then be used to identify whether the depth data corresponding to the peak is depth data from an object or depth data from a reflection. As has been explained, reflection will alter the shape of the peak (owing to the difference in subsurface scattering) in a way that can be detected by the subsurface scattering model and thus used to distinguish between objects and reflections of objects.

[0080] The type of subsurface scattering model which is applied is not particularly limited in accordance with embodiments of the disclosure. That is, any subsurface scatting model can be applied, as required, in order to identify and characterise a difference in the shape of the peaks of the depth information in a histogram of depth information acquired from the depth sensor.

[0081] In examples, the subsurface scattering model can then be defined as:

[0082]

[0083] This equtaion includes a single surface term that is not modified by an exponential decay and a single subsurface term. The temporal spread (o) is assumed to be dominated by the instrument response function such that this parameter is the same for both the direct reflection and subsurface scattering terms. Similarly, a single centre (t0) is provided as the samples are assumed to be strongly scattering. The two terms have independent amplitude parameters: Al quantifies direct surface reflection and A2 quantifies subsurface scattering.

[0084] As explained, when fitting the subsurface scattering model to the peaks in the measured histogram, the differences in the parameters which are required in order to fit the peaks can be used to classify depth measurements from objects and depth measurements from reflections (since the reflection from a reflective surface alters the shape of the peak).

[0085] While equation (1) shows an example of a subsurface scattering model which can be used in order to fit the measured peaks in the depth information acquired from the depth sensor, it will be appreciated that the present disclosure is not particularly limited in this regard. Other subsurface scattering models can be used in order to fit the measured peaks in the depth information as required, in order to detect the difference in the shape of the peak arising from the reflection of the light from a reflective surface).

[0086] Consider, now, Figure 6 of the present disclosure. Figure 6 illustrates an example of a subsurface scattering model in accordance with embodiments of the disclosure. In this example, a single peak from a histogram (such as that illustrated in Figure 4 of the present disclosure) is shown. As has been explained, in examples, the circuitry 3002 of apparatus 3000 may be configured to extract each peak from the histogram and perform identification processing on each peak to identify whether that peak corresponds to an object or a reflection of an object.In the example of Figure 6, both the extracted peak from the depth information acquired from the depth sensor and the model fit (e.g. based on the subsurface scattering model of equation 1) are shown. In order to achieve the fit to the depth information, the circuitry 3002 may be configured to perform adjustment of the parameters of the subsurface scattering model (e.g. using a known curve fitting technique (such as a least-squares fitting method)). Once the model has been fit to the depth information, the parameters of the model required to achieve the fit (and which thus characterise the shape of the peak) can then be used in order to identify whether or not the peak is a peak corresponding to an object or a reflection of an object.

[0087] In examples, the classification based on the parameters of the model can be performed by a classifier. In examples, the classifier can be a trained model, which has been trained in order to perform the classification.

[0088] In examples, the trained model can be a machine learning model (such as a deep learning model). The trained model may be a model which is trained, in advance, on a set of training data, for the purpose of training the model such that, once trained, it may perform classification of depth information acquired from the first sensor as depth information related to an object or, alternatively, depth information related to a reflection.

[0089] Training, of the trained mode, will now be described.

[0090] In examples, the methods and techniques herein may at least partly be implemented using a supervised machine learning model.

[0091] The supervised learning model is a model which is trained using labelled training data to learn a function that maps inputs (typically provided as feature vectors) to outputs (i.e. labels). The labelled training data comprises pairs of inputs and corresponding output labels. The output labels are typically provided by an operator to indicate the desired output for each input. The supervised learning model processes the training data to produce an inferred function that can be used to map new (i.e. unseen) inputs to a label.

[0092] The input data (during training and / or inference) may comprise various types of data, such as numerical values, images, video, text, or audio. In the present disclosure, the training data includes depth information which has been acquired by the depth sensor for a known type of object (i.e. either a real object or a reflection). Raw input data may be pre-processed to obtain an appropriate feature vector used as input to the model - for example, features of an image or audio input may be extracted to obtain a corresponding feature vector. It will be appreciated that the type of input data and techniques for pre-processing of the data (if required) may be selected based on the specific task the supervised learning model is used for.

[0093] As an explicit example, training data may comprise a number of different peaks of depth data for a number of different types of objects (e.g. a target object, a transparent plate, a background and a reflection) at a number of different distances from an image capture device. This training data can then be used to train the model such that it will, for previously unseen objects at previously unseen distances, be able to classify the object as a real object, a reflection, a transparent plate or a background.

[0094] In order to further improve the accuracy and efficiency with which the trained model can be used for classification, it will be appreciated that the trained model can be trained on a large library or database of depth information for different types of objects. This library or database can include multiple entries for different types of objects. Moreover, the library or database can include multipleentries for different types of reflections (e.g. different reflected objects and / or different reflective surfaces). Moreover, the library or database can include multiple entries for different types of objects under different environmental conditions (such as different temperature, humidity and the like).

[0095] Once prepared, the labelled training data set is used to train the supervised learning model. During training the model adjusts its internal parameters (e.g. weighting) so as to optimize (e.g. minimize) an error function, aiming to minimize the discrepancy between the model's predicted outputs and the labels provided as part of the training data. In some cases, the error function may include a regularization penalty to reduce overfitting of the model to the training data set.

[0096] The supervised learning model may use one or more machine learning algorithms in order to learn a mapping between its inputs and outputs. Example suitable learning algorithms include linear regression, logistic regression, artificial neural networks, decision trees, support vector machines (SVM), random forests, and the K-nearest neighbour algorithm.

[0097] In examples, the trained model may be continually updated (re-trained) as new measurements are made by the model and / or as new training data becomes available. Supervised learning can then be performed on this training data to learn a mapping between the input and corresponding outputs. Once trained, the supervised learning model may be used for inference - i.e. for predicting outputs for previously unseen input data. The supervised learning model may perform classification and / or regression tasks. In a classification task, the supervised learning model predicts discrete class labels for input data, and / or assigns the input data into predetermined categories. In a regression task, the supervised learning model predicts labels that are continuous values.

[0098] In the present disclosure, the trained model (once trained) can be used in order to perform classification of depth information acquired from the depth sensor (as being an object or a reflection of an object).

[0099] In some cases, limited amounts of labelled data may be available for training of the model (e.g. because labelling of the data is expensive or impractical). In such cases, the initially unsupervised learning model may be extended / retrained / additionally-trained with supervision to further use unlabelled data and / or to generate labelled data.

[0100] Considering using unlabelled data, the training data may comprise both labelled and unlabelled training data, and semi-supervised learning may be used to learn a mapping between the model's inputs and outputs. For example, a graph-based method such as Laplacian regularization may be used to extend a SVM algorithm to Laplacian SVM in order to perform semi-supervised learning on the partially labelled training data.

[0101] Considering generating labelled data, an active learning model may be used in which the model actively queries an information source (such as a user, or operator) to label data points with the desired outputs. Labels are typically requested for only a subset of the training data set thus reducing the amount of labelling required as compared to fully supervised learning. The model may choose the examples for which labels are requested - for example, the model may request labels for data points that would most change the current model, or that would most reduce the model's generalization error. Semi-supervised learning algorithms may then be used to train the model based on the partially labelled data set.

[0102] In this way, a trained model can be trained and used for performing classification in accordance with embodiments of the disclosure.Accordingly, the circuitry 3002 of apparatus 3000 is configured to identify from the depth data of the field of view, by a trained model, first depth data representing depth data of a reflection of an object and second depth data representing depth data of a reflective surface, the reflective surface being the surface from which the reflection of the object is received.

[0103] Consider, again, the example of Figure 5 of the present disclosure, for example. In accordance with embodiments of the disclosure, upon acquiring the depth information from the depth sensor, the circuitry 3002 may identify that the peak 5000 is a peak corresponding to an actual object in the field of view (based on the shape of the peak, as characterised through the parameters of the subsurface scattering model which has been fitted to that peak) and that the peak 5002 is a peak corresponding to a reflection of an object (based on the shape of the peak, as characterised through the parameters of the subsurface scattering model, which has been fitted to that peak).

[0104] In examples, the circuitry 3002 of apparatus 3000 may also be configured to identify that the peak 5004 is a peak corresponding to a reflective surface in the field of view of the depth sensor. In some examples, the reflective surface (or transparent plate in this specific example) may be identified in a manner similar to the identification of the reflection and the target object (i.e. through subsurface scattering). However, in other examples, one or more other techniques may also be used in order to identify a reflective surface. In some examples, the reflective plate may be identified through known plane detection techniques. For example, if a plane is detected, with some depth information from behind that plane, then it can be determined that the plane is at least partially transparent and may form a reflective surface (with the depth behind the plane potentially coming from real object and reflection of real objects, for example). Therefore, in some examples, while the circuitry 3002 of apparatus 3000 may be configured to identify objects and reflections through subsurface scattering, the circuitry 3002 of apparatus 3000 may be configured to detect the reflective surface itself through plane detection techniques known in the art. Furthermore, in some examples, if it is determined that peak 5000 is the target and peak 5002 is the reflection, it can be estimated that 5004 must be a reflective transparent surface. Therefore, in some examples, the reflective surface may be estimated based on information concerning one or more other objects which have been identified in the scene.

[0105] Nevertheless, it will be appreciated that once the circuitry 3002 of apparatus 3000 has performed the identification to identify objects, reflections and reflective surfaces from the depth information acquired from the depth sensor (or, in some examples, just the reflections and reflective surfaces), the apparatus 3000 may further be configured to perform improved location detection of an object. <Geometry Correction>

[0106] As has been explained with reference to Figure 3 of the present disclosure, circuitry 3002 of apparatus 3000 is further configured to determine location data for the object for which the reflection of the object is received based on the first depth data in relation to the second depth data, wherein the location data being indicative of a location outside a field of view.

[0107] The depth data of the field of view, as acquired from the depth sensor, is data indicative of the distance of objects within the scene from the depth sensor. This data does not distinguish between real objects and the reflections of real objects in the field of view (i.e. reflections of objects from a reflective surface, for example). Furthermore, since the data does not distinguish between real objects and reflections of real objects, the depth data for a reflected object does not necessarily show that reflected object at a correct geographical position. For example, considering the reflected object in the example of Figures 2 and 4 of the present disclosure, while the reflected object isshown at a correct distance from the depth sensor (being the distance from the depth sensor to the reflective surface plus the distance from the reflective surface to the reflected object) it is not shown, in the depth data, at correct location in the space. Instead, while, in the real world environment, the reflected object is located at a first position (as shown in Figure 2) on the same side of the reflective surface as the image capture device, in the depth data the reflected object appears to be located between the reflective surface and the target (as shown in Figure 5).

[0108] Therefore, the depth data acquired from the depth sensor cannot be used directly, in the presence of a reflective surface, to gain an understanding of the location of objects.

[0109] Rather, in embodiments of the disclosure, once the circuitry 3002 has performed identification processing in order to identify objects, reflections and reflective surfaces from the depth information acquired from the depth sensor, the circuitry 3002 of apparatus 3000 must further perform determination processing in order to determine location data for the objects. As has been described, this determination processing is performed based on the first depth data (being the depth data which has been identified as corresponding to a reflection) and the second depth data (being the depth data which has been identified as corresponding to a reflective surface).

[0110] This determination processing is made, by the circuitry 3002 of apparatus 3000, in order to adjust for the geographical distortion in the depth data arising from the presence of the reflective surface, such that an accurate location for the reflected object can be determined (even when that reflected object is outside the field of view of the depth sensor). In other words, in examples, the circuitry 3002 of apparatus 3000 may be configured to determine the location data for the object for which the reflection of the object is received based on the first depth data in relation to the second depth data by correcting the first depth data based on the second depth data.

[0111] Consider, now, Figures 7A to 7C of the present disclosure. Figures 7A to 7C illustrate an example of correcting point cloud geometry in accordance with embodiments of the disclosure.

[0112] The example point cloud which is shown in Figure 7A is the same as the point cloud which is shown in, and has been described with reference to, Figure 4 of the present disclosure. The point cloud is a visual representation of the depth data which has been acquired by the first sensor in an example situation such as that illustrated in Figure 2 of the present disclosure. Each point, in the point cloud, is a depth measurement which has been performed by the depth sensor. Clusters (or peaks) in the point cloud correspond to objects and reflections of objects within the field of view of the depth sensor.

[0113] The circuitry 3002 of apparatus 3000 has performed identification processing on this depth data, in order to identify which of the peaks correspond to actual objects, which of the peaks correspond to reflections and which of the peaks correspond to reflective surfaces. This has been performed in the manner which has been described with reference to Figures 5 and 6 of the present disclosure (e.g. based upon characteristic shapes of the respective peaks (using, a subsurface scattering model, for example)).

[0114] As such, the peak 7000 has been identified as corresponding to a real object (here, the target as shown in Figure 2), the peak 7002 has been identified as corresponding to a reflection of a real object (here, a reflection of the reflected target as shown in Figure 2) and the peak 7004 has been identified as corresponding to a reflective surface (or transparent plate; here, the polycarbonate plate as shown in Figure 2).Once the peaks have been thus identified, the circuitry 3002 of apparatus 3000 must perform processing to determine the correct location for reflected objects. That is, the depth information acquired for real objects (such as the depth information corresponding to the target) do not need to be adjusted, since they have not been interfered with by the reflective surface. Likewise, depth information for the reflective surface itself does not have to be adjusted, since the depth information from the reflected surface has not been interfered with by the reflective surface itself. Rather, it is the location of reflected objects which are misaligned in the depth information compared to the actual location of the reflected objects in the real world. Accordingly, it is the depth information of these reflected objects which must be adjusted.

[0115] Figure 7B shows an adjustment process which may be applied to the depth data of the reflected objects. In this example, once the reflected objects and the reflective surface have been identified in the manner described, an adjustment process is then applied to the depth information corresponding to the reflected objects. In embodiments of the disclosure, this adjustment is based on the relationship between the first depth data (i.e. the depth data corresponding to the reflected object) and the second depth data (i.e. the depth data corresponding to the reflective surface). More specifically, in this example, once the plane of the reflective surface is known (e.g. using a plane detection technique), the depth data of the reflected object can be transformed about the plane of the reflective surface. Indeed, in this specific example, transforming the depth data of the reflected object about the plane of the reflected surface comprises flipping the first depth data about the plane of the reflective surface. This is shown in Figure 7B.

[0116] The result of this transformation is shown in Figure 7C of the present disclosure. Here, the depth data corresponding to the reflected object has been transformed such that it is located in a correct geographical location corresponding to the actual location of that object in the real world (i.e. the corrected reflection as shown in Figure 7C).

[0117] In other words, since the location of the transparent plane is known and the identification processing has classified which depth data are reflected depth data, it is possible to correct the point cloud by moving the reflected points to their real world positions.

[0118] Correcting the depth data received from the depth sensor in this manner improves the recovery of location data for an object in the presence of a reflective surface (since objects are located in their correct real-world location). In fact, apparatus 3000 is able to recover location information for an object - based on its reflection - even when that object is located outside of the field of view. Thus, improved performance of an image capture device when a reflective surface is included within the field of view of the image capture device can be achieved.

[0119] The use of apparatus 3000 to determine location data for an object for which the reflection of an object is received, even when the object is located outside the field of view of the depth sensor, will now be described with reference to two example situations.

[0120] <Example Situation>

[0121] Consider, now, Figure 8 of the present disclosure. Figure 8 illustrates an example situation in accordance with embodiments of the disclosure.

[0122] More specifically, Figure 8 illustrates two different example situations of how apparatus 3000 can be used in order to determine location information for an object which is outside the field of view of the depth sensor.Taking the first example of Figure 8 (on the left-hand side of Figure 8), an image capture device (here, a camera) is provided in order to determine location information of objects in a scene. The camera may, in examples, be part of a SLAM system or a navigation system, for example.

[0123] The camera may be an example of a depth sensor as described with reference to Figure 3 of the present disclosure, which performs depth measurement of a portion of a scene falling within the field of view of the depth sensor through direct time-of-flight measurement, for example. That is, the camera may include a depth sensor for depth measurements (optionally, alongside an image sensor, for example).

[0124] An opaque wall is present in the environment in which the camera is located. In general, the camera is not able to perform depth measurement for the region behind the opaque wall in its line-of-sight (since this region is not visible to the camera). For example, even though an object is present, at position A, the camera is not able to perform depth measurement to the object (and does not even know the location of the object (or, indeed, that the object even exists)) since the object is located behind the wall in the line-of-sight of the camera.

[0125] This lack of information concerning the location of the object can cause problems in certain situations. For example, if the object is a moving object, since its current location is unknown, it may suddenly and unexpectedly appear from behind the wall; the object will only be known to the camera (and any system which relies upon the depth information from the camera) once it is in the line-of-sight of the camera. In the context of a navigation system, such as may be present in an autonomous vehicle, this sudden and unexpected appearance of the object may negatively impact the safety of the system (since any necessary action which may be required to avoid the object could only be taken once the object was visible to the camera).

[0126] However, in this example, a reflective surface is also present within the environment in which the camera is located. Accordingly, even though the object is not visible within the field of view of the camera, a reflection of that object may be visible to the camera on the reflective surface.

[0127] However, the mere presence of the reflective surface is not sufficient to improve the safety of the system. That is, for example, if the camera observed the reflection of the object, in the reflective surface, without realising that the reflection of the object was, in fact, a reflection, then the camera may consider that the object was actually present at the position from which the reflection of the object appears to be originating (i.e. the camera may incorrectly determine that the object was actually located at position B).

[0128] In such a situation, in the context of a navigation system, the incorrect determination of the position of the object may negatively impact the safety of the system (since the system may take an inappropriate action to avoid the object, based on an incorrect determination that the object was in fact location at position B (even though the object was actually located at position A)).

[0129] Rather, through the use of embodiments of the disclosure, improved performance of an image capture device when a reflective surface is included within the field of view of the image capture device can be achieved.

[0130] That is, the circuitry 3002 of apparatus 3000 will first acquire, from the camera, depth data of a field of view of the depth sensor. This will include depth information of the opaque wall, depth information of the reflective surface and depth information of the reflection of the object seen by the camera.Then, circuitry 3002 of apparatus 3000 is further configured to identify from the depth data of the field of view, by a trained model, first depth data representing depth data of a reflection of an object and second depth data representing depth data of a reflective surface, the reflective surface being the surface from which the reflection of the object is received. In other words, the apparatus 3000 will identify, based on the depth data which has been acquired from the camera, that the reflection of the object seen by the camera at positon B is not actually an object, but rather a reflection of an object reflected by the reflective surface.

[0131] Finally, having performed this identification, the circuitry 3002 of apparatus 3000 is then configured to determine location data for the object for which the reflection of the object is received based on the first depth data in relation to the second depth data, wherein the location data being indicative of a location outside a field of view. In other words, even though the object - being positioned at position A - is outside of the field of view of the camera, the camera (having identified that the depth data from position B is a reflection) can use the depth information concerning both the reflection of the object as seen by the camera and the depth information of the reflective surface in order to determine the actual location of the object.

[0132] Therefore, the circuitry 3002 may determine the location of the object even though that object is not within the field of view of the camera. As such, in the example of a navigation system, the camera may use the determined location of the object from the reflection of the object to take appropriate actions to avoid that object (if required) even though that object is not - at present -directly visible within the field of view of the camera. Thus, improved performance of the camera when a reflective surface is included within the field of view of the image capture device can be achieved. Indeed, in the context of a navigation, this improved performance of the camera can further improve the safety of the system (enabling appropriate actions to be taken even before the object becomes directly visible to the camera).

[0133] Consider, now, the second example of Figure 8 of the present disclosure (on the right hand side of Figure 8). This second example is similar to the first example which has been described with reference to the left hand side of Figure 8. However, in contrast to the first example, the object is not hidden behind an opaque wall. Rather, in this second example, the object is not directly visible to the camera as it is located behind the camera (and thus cannot be directly observed by the camera).

[0134] Nevertheless, similar to the first example, despite the fact that this object is not directly visible to the camera (as it is not present within the field of view of the camera) apparatus 3000 of the present disclosure can still be used in order to determine location information for the object (even though the object, being behind the camera, is outside the field of view of the camera).

[0135] In particular, the camera may observe a reflection of the object in a reflected surface. Then, apparatus 3000 may identify that the reflection is, in fact, a reflection of the object based on the characteristic shape of the depth information as has been described with reference to Figures 4 and 5 of the present disclosure). Upon performing this identification, the circuitry 3002 may then determine location data for the object for which the reflection of the object is received based on the first depth data in relation to the second depth data, wherein the location data being indicative of a location outside a field of view. Accordingly, location information for the object behind the camera can still be determined, by apparatus 3000. Again, this improves the performance of the camera when a reflective surface is included within the field of view of the image capture device.

[0136] <Filtering>The use of apparatus 3000 to provide improved recovery of location data for an object in the presence of a reflective surface has been described. In particular, the use of the reflection of an object to identify the real world location of that object, even when the real world location of the object is outside the field of view of the depth sensor, has been described.

[0137] However, in examples, based on the classification of the depth data which has been performed by the circuitry 3002, a further filtering of the depth data as acquired from the depth sensor may also be performed.

[0138] Consider, again the first example as described with reference to Figure 8 of the present disclosure. Here, it has been explained that two problems may arise when a reflective surface is present in the field of view of the depth sensor. The first of these problems is that the location of the reflected object may be incorrectly determined. The second is that the reflection of the reflected object may itself interfere with subsequent processing which is performed (e.g. such as when the apparatus 3000 is used as part of a navigation system, or a direct time of flight based autofocus system for example). Taking the first example (on the left hand side of Figure 8), if the a system observed the reflection of the object from the acquired depth information, in the reflective surface, without realising that the reflection of the object was, in fact, a reflection, then the system may consider that the object was actually present at the position from which the reflection of the object appears to be originating (i.e. the camera may incorrectly determine that the object was actually located at position B).

[0139] Alternatively, for example, the depth data of the reflective surface may, in itself, interfere with a subsequent use of the depth data for the target object.

[0140] Accordingly, in examples, the circuitry of apparatus 3000 may be further configured to perform filtering of the depth data acquired from the depth sensor, based on the classification result obtained by the identification processing performed by the circuitry 3002, in order to filter (or remove) one or more aspects of the depth data. For example, filtering of the depth data may, in examples, be performed in order to remove, from the acquired depth data, depth data of reflections and depth data of reflective surfaces, such that only the depth data of actual objects present in the real world remain within the depth data. This can further improve the accuracy and reliability of subsequent processing performed on the acquired depth data.

[0141] Consider, now, Figures 9 and 10 of the present disclosure. Figures 9 and 10 illustrates an example of point cloud filtering in accordance with embodiments of the disclosure.

[0142] In this example, the point cloud illustrated in Figure 9 is the same as the point cloud which is shown in, and has been described with reference to, Figure 4 of the present disclosure. The point cloud is a visual representation of the depth data which has been acquired by the first sensor in an example situation such as that illustrated in Figure 2 of the present disclosure. Each point, in the point cloud, is a depth measurement which has been performed by the depth sensor. Clusters (or peaks) in the point cloud correspond to objects and reflections of objects within the field of view of the depth sensor.

[0143] The circuitry 3002 of apparatus 3000 has performed identification processing on this depth data, in order to identify which of the peaks correspond to actual objects, which of the peaks correspond to reflections and which of the peaks correspond to reflective surfaces. This has been performed in the manner which has been described with reference to Figures 5 and 6 of the present disclosure (e.g. based upon characteristic shapes of the respective peaks (using, a subsurface scattering model, for example)).As such, the peak 9000 has been identified as corresponding to a real object (here, the target as shown in Figure 2), the peak 9002 has been identified as corresponding to a reflection of a real object (here, a reflection of the reflected target as shown in Figure 2) and the peak 9004 has been identified as corresponding to a reflective surface (or transparent plate; here, the polycarbonate plate as shown in Figure 2).

[0144] Once these peaks have been identified, the circuitry 3002 is configured to remove the depth data 9002 (corresponding to the reflection of a real object) and the depth data 9004 (corresponding to a reflective surface). This is shown in Figure 9B; where the depth data 9002 and depth data 9004 have been removed.

[0145] <First Example Method>

[0146] Consider, now, Figure 11 of the present disclosure. Figure 11 of the present disclosure illustrates a first example method in accordance with embodiments of the disclosure. In examples, the first example method illustrated in Figure 11 may be implemented by a first example apparatus 3000 as described with reference to Figure 3 of the present disclosure.

[0147] The example method of Figure 11 starts at step S11000 and proceeds to step S11002.

[0148] In step S11002, the method comprises acquiring, from depth detection circuitry, depth data of a field of view of the depth detection circuitry.

[0149] In step S11004, the method comprises identifying from the depth data of the field of view, by a trained model, first depth data representing depth data of a reflection of an object and second depth data representing depth data of a reflective surface, the reflective surface being the surface from which the reflection of the object is received.

[0150] Then, in step S11006, the method comprises determining location data for the object for which the reflection of the object is received based on the first depth data in relation to the second depth data, the location data being indicative of a location of the object outside the field of view.

[0151] The method then proceeds to, and ends with, step S11008.

[0152] In this way, improved recovery of location data for an object in the presence of a reflective surface can be achieved. In particular, location information for an object - based on its reflection - can be recovered, even when that object is located outside of the field of view. Thus, improved performance of an image capture device when a reflective surface is included within the field of view of the image capture device can be achieved.

[0153] It will be appreciated that the present disclosure is not particularly limited to the example method which is illustrated in Figure 11 of the present disclosure. For example, a number of the steps of the method can be performed in a different order than that illustrated in Figure 11 and / or may be performed in parallel to each other. In some examples, a number of additional method steps (not shown in this example) may also be included in the method as required. Furthermore, in some examples, one or more of the method steps of Figure 11 may be repeated as required. Therefore, the present disclosure is not particularly limited to the specific illustration of the method shown in the example of Figure 11.

[0154] <Second Example Apparatus>

[0155] Consider, now, Figure 12 of the present disclosure. Figure 12 of the present disclosure illustrates a second example apparatus in accordance with embodiments of the disclosure.The second example apparatus 12000 of Figure 12 comprises circuitry 12002.

[0156] The circuitry 12002 of apparatus 12000 is configured to receive from a target field of view, via depth detection circuitry, data indicative of depth measurements for both a first object in the field of view, through an intermediate surface having reflective properties, and a depth measurement less than that of the first object and at least reflection of an object.

[0157] The circuitry 12002 of apparatus 12000 is also configured to generate respective data indicative of the depth measurements.

[0158] The circuitry 12002 of apparatus 12000 is also configured to identify from the respective data indicative of the depth measurements, by a trained model, first depth data representing the first object in the field of view and second depth data representing the reflection of the object.

[0159] Then, circuitry 12002 of apparatus 12000 is configured to generate control signals to control focusing circuitry according to an input signal used to select one of the first depth data and the second depth data; wherein the circuitry is configured to generate the control signals to control the focusing circuitry based on a selected one of the first depth data and the second depth data.

[0160] In this way, improved focusing for an object can be performed in the presence of a reflective surface. Thus, improved performance of an image capture device when a reflective surface is included within the field of view of the image capture device can be achieved.

[0161] In examples, the circuitry 12002 may be implemented as processing circuitry such as the processing circuitry 1002 as described with reference to Figure 1 of the present disclosure. While, in the example of Figure 12, a single circuitry 12002 is shown, it will be appreciated that the present disclosure is not particularly limited in this regard. In examples, different functions provided by the circuitry 12002 may be distributed amongst one or more separate circuitry or processing units (e.g. receiving unit 12002A, identifying unit 12002B and generating unit 12002C). Therefore, it will be appreciated that the present disclosure is not particularly limited to the configuration illustrated in the example of Figure 12.

[0162] Furthermore, a number of additional components may, optionally, be provided as part of apparatus 12000 in addition to the circuitry 12002. That is, as an example, the apparatus 12000 may optionally further comprise a depth sensor 12004 (as an example of depth detection circuitry). That is, the depth detection circuitry from which the depth data indicative of depth measurements is acquired may - in some examples - be part of the apparatus 12000 itself. However, the present disclosure is not particularly limited in this regard. In other examples, the depth sensor 12004 (as an example of depth detection circuitry) may be external to, and separate from, the apparatus 12000. In this situation, the apparatus 12000 may be communicatively coupled to the depth sensor 12004 and may thus acquire the depth data from the depth sensor 12004 on this basis.

[0163] The details of the depth sensor 12004 (as an example of depth detection circuitry) are the same as have already been described in the context of the depth sensor 3004 with reference to Figure 3 of the present disclosure. Therefore, while it will be appreciated that the depth sensor 12004 may be, for example, a direct time-of-flight based sensor, further details of the depth sensor 12004 will not be provided again at this stage.

[0164] In addition, in examples, the apparatus 12000 may optionally comprise the focusing circuitry 12006. In other examples, the focusing circuitry 12006 may be external to and separate from the apparatus 12000. In this example, the apparatus 12000 may be communicatively coupled to the focusingcircuitry 12006, such that the focusing circuitry 12006 can receive the control signals from the apparatus 12000.

[0165] The focusing circuitry 12006 may, in examples, be any circuitry which, based on the control signals generated by apparatus 12000, can control focusing of an image capture device. For example, the control circuitry may be circuitry which, based on the control signal, controls a position and / or orientation of one of more focusing elements in order to change a focal position of the image capture device. However, it will be appreciated that the type of the focusing circuitry is not particularly limited in accordance with embodiments of the disclosure and may vary depending, for example, on the type of the image capture device for which the focus is being controlled.

[0166] In some examples, the image capture device may be external to, and separate from, the apparatus 12000. That is, the focusing circuitry 12006 may be communicatively coupled to an image capture device, in order that the focusing circuitry controls the focusing of that image capture device. In examples, the apparatus 12000 may be part of an image capture device. In this case, the apparatus 12000 may also comprise an image sensor for capturing an image of the scene (such as a visible light image of the scene (such as an RGB image)). In this situation, the focusing circuitry 12006 may control focusing of the image sensor of the apparatus 12000 to change a focal position of the image sensor. However, the present disclosure is not particularly limited in this regard. That is, in examples, both the focusing circuitry 12006 and the image sensor may be external to apparatus 12000 (as part of an external image capture device, for example).

[0167] In addition, optionally, the second example apparatus 12000 may also optionally comprise input circuitry 12008. The input circuitry 12008 may be used in order to receive the input signal which is used to select one of the first depth data and the second depth data. However, in other examples, the input circuitry 12008 may be external to, and separate from, the apparatus 12000. In such an example, the apparatus 12000 may be communicatively coupled to the input circuitry 12008, such that the input signal used to select one of the first depth data and the second depth data may be received. In examples, the input circuitry 12008 may be implemented as a user input device 1006 as described with reference to Figure 1 of the present disclosure. In examples, the input circuitry 12008 may enable a user to provide an input signal to select one of the first depth data and the second depth data through a user interface (such as a graphical user interface). However, the input circuitry 12008 is not particularly limited in this regard and any suitable input circuitry can be used as required depending on the situation to which the embodiments of the disclosure are applied.

[0168] Further details of the example apparatus 12000 will now be described.

[0169] <Focus Selection>

[0170] As has been explained in the Background, an image capture device (such as may be used in a wide range of technology) can struggle to perform with a desired degree of accuracy when reflections are present within the field of view of the image capture device. These reflections can interfere with the image capture performed by the image capture device. For example, if glass (or another reflective -or partially reflective - surface) is present, depth from the glass or depth from reflected objects can interfere with the process of obtaining correct depth information and focus.

[0171] Indeed, as has been explained with reference to Figures 2A and 2B of the present disclosure, the presence of a reflection in an image can cause known autofocus mechanisms, such as PDAF, to incorrectly determine the focus positon when trying to image a target.In particular, in the example of Figures 2A and 2B, when the camera is imaging the target through a partially reflective surface (the polycarbonate plate 2cm), it has been shown that an autofocus mechanism such as PDAF will incorrectly determine the focus position (see Figure 2B).

[0172] Apparatus 12000 enables improved focusing for an object can be performed in the presence of a reflective surface. Thus, improved performance of an image capture device when a reflective surface is included within the field of view of the image capture device can be achieved.

[0173] In order to achieve this, apparatus 12000 first receives from a target field of view, via depth detection circuitry (such as the depth sensor) data indicative of depth measurements for both a first object in the field of view, through an intermediate surface having reflective properties, and a depth measurement less than that of the first object and at least one reflection of an object. These depth measurements can then be used, by apparatus 12000, to generate respective depth data indicative of the depth measurements.

[0174] The depth data is information which indicates the distance of an object from the depth detection circuitry (such as the depth sensor). As has been described, the depth sensor may be a direct time-of-flight based sensor. In this situation, the depth information may be determined based on the time it takes for light emitted by an illuminator to return to the depth sensor having been reflected from objects within the field of view of the depth sensor. In some examples, the depth sensor may move, over time, in order to perform a scan of a scene (i.e. such that the field of view of the camera is changed to cover a portion of the scene which was previously outside the field of view of the depth sensor). However, in other examples, the depth sensor may remain in a fixed location (such that the field of view of the depth sensor covers a fixed portion of a scene).

[0175] Indeed, it will be appreciated that the depth data indicative of the depth measurements which is generated based on the depth measurements from the depth detection circuitry may, for example, comprise a point cloud of depth data (which can also be displayed as a histogram of depth data) such has been described with reference to Figures 4 and 5 of the present disclosure (in the context of the first example apparatus 3000). As such, further details of the depth data (including the point cloud) will not be provided again at this stage.

[0176] Once the depth data has been generated, the circuitry 12002 of apparatus 12000 is configured to identify from the respective data indicative of the depth measurements, by a trained model, first depth data representing the first object in the field of view and second depth data representing the second object in the field of view. That is, it will be appreciated that an example set-up (such as that based on the example situation of Figure 2) it will be known which peak in the depth information corresponds to which of the objects present in the scene (i.e. it is known which peak corresponds to the transparent plate, which peak corresponds to the reflected object and which peak corresponds to the target). However, more generally, such as in a real world application, it may not be known which of the peaks corresponds to which of the objects. Indeed, it may not necessarily be known whether or not a reflective surface is even present within the field of view of the depth sensor. Accordingly, in a typical situation for which an image capture device may be used (e.g. as part of a SLAM system or as part of a navigation system), the apparatus 12000 may receive distance information from the depth sensor which forms, for example, a multi-peak point cloud. However, the apparatus may not know which clusters (or peaks) in the point cloud correspond to which objects.

[0177] Accordingly, before the depth information from the depth sensor can be used in order to provide improved image capture performance of an image capture device in the presence of a reflectivesurface, it is first necessary for the circuitry 12002 of apparatus 12000 to perform identification processing in order to identify data representing a reflected object and / or a reflective surface within the field of view of the depth sensor.

[0178] As has been explained - in the context of the first example apparatus 3000 - the inventors have realised that it is possible to use changes in the way light scatters from the surface of an object to distinguish between distance measurements to an object and distance measurements to a reflection of an object. In other words, the apparatus 12000 may perform an identification process, based on the shape of the peaks in the depth data generated from the measurements performed by the depth detection circuitry, in order to perform a classification of each object (i.e. each cluster or peak in the depth data) as being either an object, a reflection and / or a reflected surface.

[0179] The details of the identification process performed by the circuitry 12002 of apparatus 12000 are the same as have already been described with reference to Figures 5 and 6 of the present disclosure, in the context of the first example apparatus 3000. Accordingly, further repetition of the details of the identification process will not be provided again at this stage.

[0180] Nevertheless, it will be appreciated that once the circuitry 12002 of apparatus 12000 has performed the identification to identify objects, reflections and reflective surfaces from the depth information acquired from the depth sensor (or, in some examples, just the reflections and reflective surfaces), the apparatus 12000 may further be configured to perform improved focusing on a target object within the field of view.

[0181] Consider, again, the example situation of Figure 2A and 2B of the present disclosure. In this example, the failure of PDAF to identify a correct focusing position for the target object arises in view of the ambiguity caused by the presence of the reflection of the reflected target within the field of view of the camera. That is, the camera does not know whether or not the reflection is actually a reflection or a real object. Accordingly, the presence of the reflection within the field of view interferes with the focusing mechanism when focusing on the target. However, with embodiments of the disclosure, the identification process which is performed enables the apparatus 12000 to identify first depth data representing the first object (e.g. the target object) in the field of view and second depth data representing the second object (e.g. the reflection of the reflected object) in the field of view.

[0182] That is, apparatus 12000 is able to disambiguate the depth data corresponding to the different objects within the field of view.

[0183] At this stage, the circuitry 12002 of apparatus 12000 is configured to generate control signals to control the focusing circuitry according to an input signal used to select one of the first depth data and one of the second depth data. That is, as has been described with reference to Figure 12 of the present disclosure, input circuitry 12008 may be used in order to receive the input signal which is used to select one of the first depth data and the second depth data. For example, a user may be provided with information concerning the different objects which have been identified, and then may be asked to provide a selection concerning a specific object from amongst those objects as an object which is to be focused on by focusing circuitry of an image capture device.

[0184] Consider the example situation of Figure 2 of the present disclosure (and the corresponding point cloud of depth data as illustrated in Figure 4 of the present disclosure). In this example, there are three different potential objects within the field of view of the camera. The first of these is the target (behind the polycarbonate plate), the second is the reflection of the reflected object (which, in thedepth data, appears at a distance between the polycarbonate plate and the target) and thirdly the polycarbonate plate itself.

[0185] Having performed the identification process on the depth data, the circuitry 12002 of apparatus 12000 is able to classify the depth data corresponding to each of these respective objects (e.g. as shown in Figure 7A, the circuitry 12002 of apparatus 12000 is able to classify the depth data 7000 as the target, the depth data 7002 as the reflection of the reflected object and depth data 7004 as the transparent polycarbonate plate).

[0186] In examples, the different objects which have been identified (the target, reflection and transparent plate in this example) may then be presented to a user, such that the user can make a selection of which of these objects to focus on. In examples, the different objects which have been identified may be presented to the user via a user interface. In examples, the user interface may be a graphical user interface. In examples, the user may then make a selection of one of the objects which has been identified via a user input. The user input may be provided through any suitable user input device. For example, the user may provide an input such as a touch input, a button press, a voice command or the like in order to select one of the objects which has been identified. For example, the user may wish to obtain an image of the target. In this example, the user may select the target from the different objects which have been identified. Alternatively, for example, the user may wish to obtain an image of the reflection of the reflected object. Accordingly, the user may then select the reflection of the reflected object from the different objects which have been identified.

[0187] Once this input selection has been made, the circuitry 12002 is then configured to generate control signals to control the focusing circuitry, wherein the generated control signals control the circuitry based on the selection of the object which has been made.

[0188] For example, if the user has selected the target 7000 for image capture, the circuitry 12002, knowing the depth data which corresponds to this target, may generate control signals to cause focusing circuitry of the image capture device to focus on this target. In the example situation of Figure 2, the depth data of target indicates that the target is a distance of 1.6m from the camera. Accordingly, the circuitry 12002 may generate control signals to cause focusing circuitry of the image capture device to focus on this target; for example, the control signal may cause the focusing circuitry of the image capture device to move the lens position of a lens of the image capture device to position 185. Importantly, since a selection has been made from amongst the different objects which have been identified from the depth data, the focusing can be performed specifically for the selected object. In other words, depth data from other objects (being objects other than the selected object) within the field of view is disregarded and is not further considered in the context of focusing on the selected object.

[0189] It will be appreciated that the type of the control signals generated are not particularly limited and will vary depending on the specific situation to which the embodiments of the disclosure are applied (e.g. the type of focusing mechanism present in the image capture device, for example).

[0190] Accordingly, interference from other objects within the field of view (other than the selected object) is reduced, such that improved focusing for an object can be performed in the presence of a reflective surface.

[0191] <Filtering>

[0192] Consider, now, Figure 13 of the present disclosure. Figure 13 illustrates an example process flow in accordance with embodiments of the disclosure. In particular, Figure 13 illustrates a high-levelschematic of a filtering pipeline which may be used, by apparatus 12000, in order to further improve focusing on a selected object. That is, the filtering pipeline illustrated in the example of Figure 13 of the present disclosure may be used, by apparatus 12000, once a selection of an object has been received by the circuitry 12002, in order to remove other objects from the depth data (and thus further minimize their impact on the focusing of the image capture device).

[0193] In the example of Figure 13, the circuitry 12002 of apparatus 12000 has received a selection of the target as the object which is to be focused upon. However, while this example is described with reference to this example situation, it will be appreciated that the present disclosure is not particularly limited in this regard. That is, a similar filtering process may be applied by the circuitry 12002 upon the selection of any other object within the field of view (such as a selection of the reflection of the reflected object as the object which is to be focused upon).

[0194] At a first stage of this filtering pipeline, stage 13000, the circuitry 12002 of apparatus 12000 is first configured to perform reflection removal. This is based upon the identification process which has been made, which classifies the depth data which has been received as corresponding to a real object or a reflection. For example, in context of the example situation shown in Figure 2, the circuitry 12002 of apparatus 12000 may have classified the depth data 7002 as shown in Figure 7A as depth data corresponding to a reflection (based on the shape of the peak of the histogram, as has already been described with reference to Figure 5 of the present disclosure). Accordingly, if a selection has been made to focus on the target, the circuitry 12002 may perform filtering to remove all those data points which have been classified as corresponding to a reflection from the depth data.

[0195] Then, in the second stage of the filtering pipeline, stage 13002, the circuitry 12002 of apparatus 12000 is then configured to remove the transparent plate (or reflective surface) from the depth data. That is, since the selection has been made to focus on the target, the circuitry 12002 may then remove the reflective surface from the depth data (in order to further minimize its impact on the focusing of the image capture device). In examples, the depth data corresponding to the reflective surface may have been identified by the identification process which has been performed. In examples, the depth data corresponding to the reflective surface may be identified using plane detection techniques. For example, glass (on another transparent material) may be assumed if a plane is detected with depth behind it. Accordingly, if a selection has been made to focus on the target, the circuitry 12002 may perform filtering to remove all those data points which have been classified as corresponding to a reflective surface from the depth data.

[0196] Finally, in a third stage of the filtering pipeline, stage 13004, the circuitry 12002 of apparatus 12000 may perform inpainting of the depth data. In other words, the reflection of the reflected target and the reflective surface which were present within the depth data may have, at least partially, been obscuring the target. Accordingly, when the depth data corresponding to the reflection of the reflected target and the reflective surface have been removed, one or more gaps in the depth data corresponding to the target may be visible (i.e. there may be missing depth data corresponding to the target). This missing depth data may, in principle, interfere with the focusing performed to focus on the target.

[0197] The inpainting process may be performed using any known interpolation technique known in the art (such as an image interpolation technique, which will estimate the value of the missing depth data based on the neighbouring depth data).In this way, the filtering pipeline of Figure 13 (or any single stage thereof) may be used in order to further improve the focusing for an object that can be performed when a reflective surface is present in the field of view of the image capture device.

[0198] Consider, now, Figures 14 and 15 of the present disclosure. Figures 14 and 15 illustrate example point cloud filtering in accordance with embodiments of the disclosure.

[0199] More specifically, Figure 14 provides an example of filtering and inpainting which may be applied to depth data when the target has been selected as the object to be focused on.

[0200] The top panel 14000 of Figure 14 shows the depth data before the filtering pipeline described with reference to Figure 13 of the present disclosure has been applied. In this example, the point cloud of depth data has been projected into a 2D plane, prior to the application of the focusing pipeline. This may be performed, for example, once the selection of an object has been received from the user. In the top panel 14000 of Figure 14, the transparent plate and the reflection are still visible within the depth data. As can be seen, the depth data of the transparent plate and the depth data of the reflection at least partially obscure the depth data of the selected target. This can make it difficult to generate control signals to control the focusing circuitry to focus on the selected target.

[0201] The middle panel 14002 of Figure 14 shows the depth data once the filtering (stage 13000 and stage 13002 of the pipeline) has been applied (i.e. the filtering to remove the transparent plate and the reflection from the depth data). While the filtering to remove the depth data of the transparent plate and the depth data of the reflection has improved the visibility of the depth data of the target, it will be appreciated that a number of holes exist in the depth data.

[0202] The bottom panel 14004 of Figure 14 shows the depth data once the inpainting (stage 13004 of the pipeline) has been applied. As can be seen through a comparison of the middle panel 14002 and the bottom panel 14004, the inpainting process has filed the gaps in the depth data (arising from the missing depth data of the target).

[0203] In this way, by projecting the point cloud back into a 2D plane, filtering and inpainting the missing pixels, improved depth data (also known as a depth map) which is usable for focusing on the target can be recovered. In other words, by applying the filtering pipeline described with reference to Figure 13 to the depth data, it becomes possible to more clearly see the depth data of the target (located behind the reflection and the transparent plate) in the depth data.

[0204] Thus, improved focusing for the selected object, using the recovered depth data, can be achieved. Figure 15 of the present disclosure demonstrates an alternative situation, whereby the user has selected the reflection of the reflected object as the focus point. That is, the user has chosen to focus on the reflection (and not the target located behind the transparent plane).

[0205] Similar to the top panel 14000 of Figure 14, the top panel 15000 of Figure 15 shows the depth data -once projected into 2D - prior to the application of the filtering pipeline. That is, the transparent plate and the reflection are still visible in the depth data.

[0206] The middle panel 15004 of Figure 15 shows the depth data once the filtering (stage 13000 and stage 13002 of the pipeline) has been applied (i.e. the filtering to remove the transparent plate and the target from the depth data). While the filtering to remove the depth data of the transparent plate and the depth data of the target has improved the visibility of the depth data of the reflection, it will be appreciated that a number of holes exist in the depth data.The bottom panel 15004 of Figure 15 shows the depth data once the inpainting (stage 13004 of the pipeline) has been applied. As can be seen through a comparison of the middle panel 15002 and the bottom panel 15004, the inpainting process has filed the gaps in the depth data (arising from the missing depth data of the target).

[0207] In this way, by projecting the point cloud back into a 2D plane, filtering and inpainting the missing pixels, improved depth data (also known as a depth map) which is usable for focusing on the reflection (as the selected object) can be recovered. In other words, by applying the filtering pipeline described with reference to Figure 13 to the depth data, it becomes possible to more clearly see the depth data of the reflection in the depth data (e.g. without the influence of the reflective surface). Thus, improved focusing for the selected object, using the recovered depth data, can be achieved. While the example of Figure 13 of the present disclosure has been described as a single pipeline, it will be appreciated that the present disclosure is not particularly limited in this regard. In examples, individual stages of the pipeline may be applied to the depth data. In examples, the pipeline may be modified such that a number of the different stages of the pipeline are repeated a number of times as required. In examples, the pipeline may be applied as a linear process. However, in examples, a number of the different stages of the pipeline may be performed in parallel. Indeed, the order of the different stages of the pipeline is not limited to that shown in Figure 13: a number of the different stages of the pipeline of Figure 13 may be performed in a different order to that illustrated in Figure 13 of the present disclosure if required.

[0208] Furthermore, while the example filtering pipeline of Figure 13 has been described with reference to an example situation such as that described with reference to Figure 2 of the present disclosure (and the specific examples of Figures 14 and 15 in this context) it will be appreciated that the present disclosure is not particularly limited in this regard, with the filtering pipeline being applicable, more generally, to any example situation to which the embodiments of the disclosure are applied.

[0209] <Second Example Method>

[0210] Consider, now, Figure 16 of the present disclosure. Figure 16 of the present disclosure illustrates a second example method in accordance with embodiments of the disclosure. In examples, the second example method illustrated in Figure 16 may be implemented by a second example apparatus 12000 as described with reference to Figure 12 of the present disclosure.

[0211] The example method of Figure 16 starts at step S16000 and proceeds to step S16002.

[0212] In step S16002, the method comprises receiving from a target field of view, via depth detection circuitry, data indicative of depth measurements for both a first object in the field of view, through an intermediate surface having reflective properties, and a depth measurement less than that of the first object and at least one reflection of an object.

[0213] Then, in step S16004, the method comprises generating respective data indicative of the depth measurements.

[0214] In step S16006, the method comprises identifying from the respective data indicative of the depth measurements, by a trained model, first depth data representing the first object in the field of view and second depth data representing the second object in the field of view.

[0215] In step S16008, the method comprises generating control signals to control focusing circuitry according to an input signal used to select one of the first depth data and the second depth data;

[0216] 1wherein the circuitry is configured to generate the control signals to control the focusing circuitry based on a selected one of the first depth data and the second depth data.

[0217] The method then proceeds to, and ends with, step S16010.

[0218] In this way, improved focusing for an object can be performed in the presence of a reflective surface. Thus, improved performance of an image capture device when a reflective surface is included within the field of view of the image capture device can be achieved.

[0219] It will be appreciated that the present disclosure is not particularly limited to the example method which is illustrated in Figure 16 of the present disclosure. For example, a number of the steps of the method can be performed in a different order than that illustrated in Figure 16 and / or may be performed in parallel to each other. In some examples, a number of additional method steps (not shown in this example) may also be included in the method as required. Furthermore, in some examples, one or more of the method steps of Figure 16 may be repeated as required. Therefore, the present disclosure is not particularly limited to the specific illustration of the method shown in the example of Figure 16.

[0220] <Third Example Apparatus>

[0221] Consider, now, Figure 17 of the present disclosure. Figure 17 of the present disclosure illustrates a third example apparatus 17000 in accordance with embodiments of the disclosure.

[0222] The example apparatus 17000 of Figure 17 comprises circuitry 17002.

[0223] The circuitry 17002 is configured to receive from a target field of view, via depth detection circuitry, data indicative of depth measurements for both a first object in the field of view, through an intermediate surface having reflective properties, and a depth measurement less than that of the first object and at least one reflection of an object.

[0224] Furthermore, the circuitry 17002 is configured to generate respective data indicative of the depth measurements.

[0225] The circuitry 17002 is also configured to identify from the respective data indicative of the depth measurements, by a trained model, first depth data representing the first object in the field of view and second depth data representing the reflection of the object.

[0226] Then, the circuitry 17002 is configured to perform phase detection auto focus to determine a temporary focus point in the field of view.

[0227] Circuitry 17002 is also configured to select one of the first depth data and the second depth data based on a comparison of the first depth data and second depth data with the temporary focus point from the phase detection auto focus.

[0228] Finally, circuitry 17004 is configured to generate control signals to control focusing circuitry according to the selected one of the first depth data and the second depth data.

[0229] In this way, improved focusing for an object can be performed in the presence of a reflective surface. Thus, improved performance of an image capture device when a reflective surface is included within the field of view of the image capture device can be achieved.

[0230] In examples, the circuitry 17002 may be implemented as processing circuitry such as the processing circuitry 1002 as described with reference to Figure 1 of the present disclosure. While, in the example of Figure 17, a single circuitry 17002 is shown, it will be appreciated that the presentdisclosure is not particularly limited in this regard. In examples, different functions provided by the circuitry 17002 may be distributed amongst one or more separate circuitry or processing units (e.g. receiving unit 17002A, identifying unit 17002B, performing unit 17002C, selecting unit 17002D and generating unit 17002E). Therefore, it will be appreciated that the present disclosure is not particularly limited to the configuration illustrated in the example of Figure 17.

[0231] Furthermore, a number of additional components may, optionally, be provided as part of apparatus 17000 in addition to the circuitry 17002. That is, as an example, the apparatus 17000 may optionally further comprise a depth sensor 17004 (as an example of depth detection circuitry). That is, the depth detection circuitry from which the depth data indicative of depth measurements is acquired may - in some examples - be part of the apparatus 17000 itself. However, the present disclosure is not particularly limited in this regard. In other examples, the depth sensor 17004 (as an example of depth detection circuitry) may be external to, and separate from, the apparatus 17000. In this situation, the apparatus 17000 may be communicatively coupled to the depth sensor 17004 and may thus acquire the depth data from the depth sensor 17004 on this basis.

[0232] The details of the depth sensor 17004 (as an example of depth detection circuitry) are the same as have already been described in the context of the depth sensor 3004 with reference to Figure 3 of the present disclosure and the depth sensor 12004 with reference to Figure 12 of the present disclosure. Therefore, while it will be appreciated that the depth sensor 17004 may be, for example, a direct time-of-flight based sensor, further details of the depth sensor 17004 will not be provided again at this stage.

[0233] In addition, in examples, the apparatus 17000 may optionally comprise the focusing circuitry 17006. In other examples, the focusing circuitry 17006 may be external to and separate from the apparatus 17000. In this example, the apparatus 17000 may be communicatively coupled to the focusing circuitry 17006, such that the focusing circuitry 17006 can receive the control signals from the apparatus 17000.

[0234] In some examples, the image capture device may be external to, and separate from, the apparatus 17000. That is, the focusing circuitry 17006 may be communicatively coupled to an image capture device, in order that the focusing circuitry controls the focusing of that image capture device. In examples, the apparatus 17000 may be part of an image capture device. In this case, the apparatus 17000 may also comprise an image sensor for capturing an image of the scene (such as a visible light image of the scene (such as an RGB image)). In this situation, the focusing circuitry 17006 may control focusing of the image sensor of the apparatus 17000 to change a focal position of the image sensor. However, the present disclosure is not particularly limited in this regard. That is, in examples, both the focusing circuitry 12006 and the image sensor may be external to apparatus 17000 (as part of an external image capture device, for example). The image sensor may, as previously described, be an image sensor configured for PDAF focusing (i.e. where one or more pixels of the image sensor are used for phase detection).

[0235] However, it will be appreciated that the details of the focusing circuitry 17006 are the same as have already been described in the context of the focusing circuitry 12006 with reference to Figure 12 of the present disclosure. Therefore, further details of the focusing circuitry 17006 will not be provided again at this stage.

[0236] Further details of the apparatus 17000 will now be described.

[0237] <Automatic Focus Selection>As has been explained in the Background, an image capture device (such as may be used in a wide range of technology) can struggle to perform with a desired degree of accuracy when reflections are present within the field of view of the image capture device. These reflections can interfere with the image capture performed by the image capture device. For example, if glass (or another reflective -or partially reflective - surface) is present, depth from the glass or depth from reflected objects can interfere with the process of obtaining correct depth information and focus.

[0238] Indeed, as has been explained with reference to Figures 2A and 2B of the present disclosure, the presence of a reflection in an image can cause known autofocus mechanisms, such as PDAF, to incorrectly determine the focus positon when trying to image a target.

[0239] In particular, in the example of Figures 2A and 2B, when the camera is imaging the target through a partially reflective surface (the polycarbonate plate 2cm), it has been shown that an autofocus mechanism such as PDAF will incorrectly determine the focus position (see Figure 2B).

[0240] Apparatus 12000, which has been described with reference to Figure 12 of the present disclosure enables a user to select an object, from the different objects which have been identified through the identification process, as an object which should be focused upon. Filtering and inpainting can then be performed, once the object has been selected, in order to improve the focusing of an image capture device on the selected object.

[0241] However, apparatus 17000 provides an automatic focus selection on an object, based on the different objects which have been identified. Thus, improved performance of an image capture device when a reflective surface is included within the field of view of the image capture device can be achieved.

[0242] In order to achieve this, apparatus 17000 first receives, from a target field of view, via depth detection circuitry (such as the depth sensor) data indicative of depth measurements for both a first object in the field of view, through an intermediate surface having reflective properties, and a depth measurement less than that of the first object and at least one reflection of an object. These depth measurements can then be used, by apparatus 17000, to generate respective depth data indicative of the depth measurements.

[0243] The depth data is information which indicates the distance of an object from the depth detection circuitry (such as the depth sensor). As has been described, the depth sensor may be a direct time-of-flight based sensor. In this situation, the depth information may be determined based on the time it takes for light emitted by an illuminator to return to the depth sensor having been reflected from objects within the field of view of the depth sensor. In some examples, the depth sensor may move, over time, in order to perform a scan of a scene (i.e. such that the field of view of the camera is changed to cover a portion of the scene which was previously outside the field of view of the depth sensor). However, in other examples, the depth sensor may remain in a fixed location (such that the field of view of the depth sensor covers a fixed portion of a scene).

[0244] Indeed, it will be appreciated that the depth data indicative of the depth measurements which is generated based on the depth measurements from the depth detection circuitry may, for example, comprise a point cloud of depth data (which can also be displayed as a histogram of depth data) such has been described with reference to Figures 4 and 5 of the present disclosure (in the context of the first example apparatus 3000). As such, further details of the depth data (including the point cloud) will not be provided again at this stage.Once the depth data has been generated, the circuitry 17002 of apparatus 17000 is configured to identify from the respective data indicative of the depth measurements, by a trained model, first depth data representing the first object in the field of view and second depth data representing the second object in the field of view. That is, it will be appreciated that an example set-up (such as that based on the example situation of Figure 2) it will be known which peak in the depth information corresponds to which of the objects present in the scene (i.e. it is known which peak corresponds to the transparent plate, which peak corresponds to the reflected object and which peak corresponds to the target). However, more generally, such as in a real world application, it may not be known which of the peaks corresponds to which of the objects. Indeed, it may not necessarily be known whether or not a reflective surface is even present within the field of view of the depth sensor. Accordingly, in a typical situation for which an image capture device may be used (e.g. as part of a SLAM system or as part of a navigation system), the apparatus 17000 may receive distance information from the depth sensor which forms, for example, a multi-peak point cloud. However, the apparatus may not know which clusters (or peaks) in the point cloud correspond to which objects.

[0245] Accordingly, before the depth information from the depth sensor can be used in order to provide improved image capture performance of an image capture device in the presence of a reflective surface, it is first necessary for the circuitry 17002 of apparatus 17000 to perform identification processing in order to identify data representing a reflected object and / or a reflective surface within the field of view of the depth sensor.

[0246] As has been explained - in the context of the first example apparatus 3000 (and, also, the second example apparatus 12000) - the inventors have realised that it is possible to use changes in the way light scatters from the surface of an object to distinguish between distance measurements to an object and distance measurements to a reflection of an object. In other words, the apparatus 17000 may perform an identification process, based on the shape of the peaks in the depth data generated from the measurements performed by the depth detection circuitry, in order to perform a classification of each object (i.e. each cluster or peak in the depth data) as being either an object, a reflection and / or a reflected surface.

[0247] The details of the identification process performed by the circuitry 17002 of apparatus 17000 are the same as have already been described with reference to Figures 5 and 6 of the present disclosure, in the context of the first example apparatus 3000. Accordingly, further repetition of the details of the identification process will not be provided again at this stage.

[0248] Nevertheless, it will be appreciated that once the circuitry 17002 of apparatus 17000 has performed the identification to identify objects, reflections and reflective surfaces from the depth information acquired from the depth sensor (or, in some examples, just the reflections and reflective surfaces), the apparatus 17000 may further be configured to perform improved focusing on a target object within the field of view using an automatic processing technique.

[0249] Consider, again, the example situation of Figure 2A and 2B of the present disclosure. In this example, the failure of PDAF to identify a correct focusing position for the target object arises in view of the ambiguity caused by the presence of the reflection of the reflected target within the field of view of the camera. That is, the camera does not know whether or not the reflection is actually a reflection or a real object. Accordingly, the presence of the reflection within the field of view interferes with the focusing mechanism when focusing on the target. However, with embodiments of the disclosure, the identification process which is performed enables the apparatus 17000 to identify first depth data representing the first object (e.g. the target object) in the field of view and seconddepth data representing the second object (e.g. the reflection of the reflected object) in the field of view.

[0250] That is, apparatus 17000 is able to disambiguate the depth data corresponding to the different objects within the field of view. On this basis, apparatus 17000 is then able to perform automatic focusing in order to focus on an object, even when a reflection is present within the field of view. Consider, now, Figure 18 of the present disclosure. Figure 18 illustrates an example situation in accordance with embodiments of the disclosure.

[0251] More specifically, Figure 18 illustrates an example situation whereby apparatus 17000 is used in order to provide automatic focusing on an object from amongst a plurality of objects present in a scene. In this example, apparatus 17000 is applied to an example situation such as that which has been described with reference to Figure 2 of the present disclosure. That is, in this example, a first object (the target) is provided at a first distance from the camera (1.6m in this example), with a reflective surface (a polycarbonate plate of 2cm thickness) being provided between the camera and the target. The polycarbonate plate is an intermediate surface, in this regard, since it is arranged between the camera and the target (such that the camera views the target through this surface). In this example, the polycarbonate plate is at a distance of approximately 0.3m from the camera. Furthermore, a reflected target object is also provided, the reflected object being on the same side of the polycarbonate plate as the camera. The reflected target is arranged at a distance of approximately 0.4m from the polycarbonate plate. The camera, when viewing the target, will see a reflection of the reflected target on the polycarbonate plate.

[0252] In a first stage 18000 of the example of Figure 18, the circuitry 17002 performs identification processing, on the depth data, in order to identify the objects present in the scene (and thus classify the depth data on this basis). For example, as illustrated in Figure 7A of the present disclosure, the apparatus 17000 may classify the depth data 7000 as the target, the depth data 7002 as the reflection and the depth data 7004 as the transparent plate. This identification and classification process can be performed, for example, as described with reference to Figure 5 of the present disclosure (i.e. based on the shape of the peak of the depth data).

[0253] Once this identification process has been completed, the circuitry 17002 of apparatus 17000 is then configured to determine a lens position (i.e. a focus position) for each object which has been identified. Notably, at this stage, processing is not performed in order to generate control signals to control the focusing circuitry to perform focusing on this basis. Rather, a number of different candidate lens positions are determined (each corresponding to an appropriate focus position for a given object).

[0254] Here, in this example, a lens position is determined for the target object (which has been identified at a distance of 1.6m from the camera) and a lens position is determined for the reflection (with the reflected target being located at a total distance of 0.7m from the target). These two lens positions are two candidate lens positions (one corresponding to each object). However, at this stage, it is not known which of these candidate lens positions should actually be used in order to control the focusing circuitry (i.e. which of the objects (the target or the reflection) should be focused upon). Next, in a second stage 18002 of the example of Figure 18, an alternative focus mechanism is performed. For example, a PDAF focus may be performed by the circuitry of apparatus 17000. This PDAF processing is performed based on a visible light image acquired by a camera or image sensor (e.g. an RGB image, for example).Notably, while the PDAF focus may be performed by the circuitry of apparatus 17000, the camera (or image sensor) which is used in order to capture the image data which is used for performing the PDAF focus may, itself, be external to and separate from the apparatus 17000. That is, the apparatus 17000 may acquire information from the external camera (or image sensor) and may then perform PDAF focusing using this acquired information. In examples, the apparatus 17000 may be part of a camera, such that the camera and the apparatus form a single device in this respect. In examples, the apparatus 17000 may comprise the image sensor which is used for the PDAF focusing.

[0255] The use of the PDAF, in combination with the focus position as determined in the first stage 18000 (i.e. based on the depth data (such as direct time-of-flight data)) can enable the apparatus 17000 to make a selection of which of the plurality of objects should be the object that is focused upon.

[0256] Consider a situation where there is no reflection in the field of view. Here, the target object will be clearly visible in the RGB image. Indeed, since there is no reflection in the field of view, the reflection will not interfere with the PDAF. As such, the autofocus performed by PDAF for the target (i.e. the determined lens position) will be close to the corresponding focus performed in the first stage 18000. In this case, a comparison between the lens position determined in the first stage 18000 and the lens position determined in the second stage 18002 of Figure 18 will show that the target is the object which should be focused upon, and control signals to control the focusing circuitry to focus on the target can then be generated accordingly.

[0257] Alternatively, consider a situation where there is a reflection in the field of view, but the target (the object behind the polycarbonate plate) is not visible within the field of view. In this situation, the autofocus performed by PDAF for the reflection (i.e. the determined lens position) will be close to the corresponding focus performed in the first stage 18000. In this case, a comparison between the lens position determined in the first stage 18000 and the lens position determined in the second stage 18002 of Figure 18 will show that the target is the object which should be focused upon, and control signals to control the focusing circuitry to focus on the target can then be generated accordingly.

[0258] In other words, when only the target or only the reflection is visible in the field of view, the PDAF lens position will be very close to the lens position determined from the depth data (e.g. based on the direct time-of-flight data).

[0259] However, this situation does not occur when both the target and reflection are visible in the field of view of the visible light image (e.g. the RGB image) from the camera (as shown in the third stage 18004 of Figure 18). That is, when both the target and the reflection are visible in the field of view of the camera (in the visible light image) the target and the reflection interfere with each other, such that PDAF cannot appropriately determine a focus position. This has been described with reference to Figures 2A and 2B of the present disclosure.

[0260] However, the inventors have realised that PDAF lens position which is determined in this situation (i.e. when both the target and reflection are visible in the image) can nevertheless be used in order to make a selection from amongst the plurality of objects present as to which object to focus upon (with the lens position determined from the depth data corresponding to the selected object then being used to control the focusing circuitry).

[0261] That is, the PDAF lens position which is determined in this situation (when both the target and reflection are visible in the image) will fall between the ideal focus position for the target and the ideal focus position for the reflection (or, that is, between the focus position for the target and the focus position for the reflection which were determined in the first stage 18000).The actual lens position determined from PDAF in this situation will depend upon the relative strength of the signal from the target and the signal from the reflection in the image. When the reflection is stronger in the visible image, the PDAF lens position will be closer to the ideal lens position for the reflection, while when the target is stronger in the visible image, the PDAF lens position will be closer to the ideal lens position for the target. The relative strength of the target and the reflection in the visible light image (e.g. the RGB image) may vary depending on a number of different factors such as the relative distance of the target and the reflected object from the reflective surface, the light conditions (on both the reflected object and the target respectively), the reflective properties of the reflective surface and the like. Nevertheless, it will be appreciated that the lens position determined from PDAF in this mixed signal situation will be biased towards the ideal lens position for the dominant object in the image.

[0262] It will be appreciated that while the PDAF lens position is biased towards the ideal lens positon for the dominant object in the image, it does not match the ideal lens position for that object.

[0263] Therefore, if the PDAF lens position was used in order to generate the control signals to directly control the focusing circuitry, the level of focus achieved would be poor (with the object appearing out of focus). Nevertheless, a comparison between the PDAF lens position and the lens position determined in the first stage 18000 of Figure 18 (i.e. the lens position determined from the depth data) enables the circuitry 18000 to determine which is the dominant object in the (mixed) image and thus which object should be selected to be focused upon.

[0264] When the PDAF lens position is closer to the lens position which has been determined, in the first stage 18000, for the target, the target will be selected as the object which should be focused upon. On the other hand, when the PDAF lens position is closer to the lens position which has been determined, in the first stage 18000, for the reflection, the reflection will be selected as the object which should be focused upon. Once this selection has been made, the lens position determined from the first stage 18000 for the selected object will then be used in order to generate the control signal to control the focusing circuitry. For example, when the target has been selected as the object which should be focused upon, the lens position of the target as determined during the first stage 18000 will be used in order to generate the control signals to control the focusing circuitry.

[0265] In other words, the object to be focused upon can be determined from a comparison of the PDAF lens position with the lens position determined from the depth data, with the lens positon determined from the depth data then being used in order to generate the control signal to control the focusing circuitry. As such, focusing will automatically be performed on the dominate object in the image (even in the situation of a target and a reflection) thus enabling reliable focusing to be performed, automatically, for an object in the image (even in the presence of a reflection). Thus, improved performance of an image capture device when a reflective surface is included within the field of view of the image capture device can be achieved.

[0266] <Filtering>

[0267] It will be appreciated that the filtering pipeline of Figure 13 of the present disclosure can be applied equally be the circuitry 17002 of apparatus 17000 as has been described with reference to apparatus 12000 in order to provide improved focusing for the selected object, as required. Further details of the filtering pipeline will not be provided again at this stage.

[0268] <Third Example Method>

[0269] Consider, now, Figure 19 of the present disclosure. Figure 19 of the present disclosure illustrates a third example method in accordance with embodiments of the disclosure. In examples, the thirdexample method illustrated in Figure 19 may be implemented by a third example apparatus 17000 as described with reference to Figure 17 of the present disclosure.

[0270] The example method of Figure 19 starts at step S19000 and proceeds to step S19002.

[0271] In step S19002, the method comprises receiving from a target field of view, via depth detection circuitry, data indicative of depth measurements for both a first object in the field of view, through an intermediate surface having reflective properties, and a depth measurement less than that of the first object and at least one reflection of an object.

[0272] In step S19004, the method comprises generating respective data indicative of the depth measurements.

[0273] In step S19006, the method comprises identifying from the respective data indicative of the depth measurements, by a trained model, first depth data representing the first object in the field of view and second depth data representing the reflection of the object.

[0274] Then, in step S19008, the method comprises performing phase detection auto focus to determine a temporary focus point in the field of view.

[0275] In step S19010, the method comprises selecting one of the first depth data and the second depth data based on a comparison of the first depth data and second depth data with the temporary focus point from the phase detection auto focus.

[0276] In step S19012, the method comprises generating control signals to control focusing circuitry according to the selected one of the first depth data and the second depth data.

[0277] The method then proceeds to, and ends with, step S19014.

[0278] In this way, improved focusing for an object can be performed in the presence of a reflective surface. Thus, improved performance of an image capture device when a reflective surface is included within the field of view of the image capture device can be achieved.

[0279] It will be appreciated that the present disclosure is not particularly limited to the example method which is illustrated in Figure 19 of the present disclosure. For example, a number of the steps of the method can be performed in a different order than that illustrated in Figure 19 and / or may be performed in parallel to each other. In some examples, a number of additional method steps (not shown in this example) may also be included in the method as required. Furthermore, in some examples, one or more of the method steps of Figure 19 may be repeated as required. Therefore, the present disclosure is not particularly limited to the specific illustration of the method shown in the example of Figure 19.

[0280] <Computer Program>

[0281] Furthermore, it will be appreciated that the methods of the present disclosure (such as the first, second and third example methods) may be carried out on conventional hardware (such as that described previously herein) suitably adapted as applicable by software instruction or by the inclusion or substitution of dedicated hardware. Thus, the required adaptation to existing parts of a conventional equivalent device may be implemented in the form of a computer program product comprising processor implementable instructions stored on a non-transitory machine-readable medium such as a floppy disk, optical disk, hard disk, PROM, RAM, flash memory or any combination of these or other storage media, or realized in hardware as an ASIC (application specific integrated circuit) or an FPGA (field programmable gate array) or other configurable circuit suitable to use inadapting the conventional equivalent device. Separately, such a computer program may be transmitted via data signals on a network such as an Ethernet, a wireless network, the Internet, or any combination of these or other networks.

[0282] <Clauses>

[0283] Embodiments of the present disclosure can also be arranged in accordance with the following numbered clauses:

[0284] 1) An apparatus for determining location data for an object, the apparatus comprising circuitry configured to:

[0285] acquire, from depth detection circuitry, depth data of a field of view of the depth detection circuitry;

[0286] identify from the depth data of the field of view, by a trained model, first depth data representing depth data of a reflection of an object and second depth data representing depth data of a reflective surface, the reflective surface being the surface from which the reflection of the object is received; and

[0287] determine location data for the object for which the reflection of the object is received based on the first depth data in relation to the second depth data, wherein the location data being indicative of a location outside a field of view.

[0288] 2) The apparatus according to clause 1, wherein the depth detection circuitry a is direct time-of-flight based sensor.

[0289] 3) The apparatus according to clause 1 or 2, wherein the depth data comprises a multi-peak point cloud.

[0290] 4) The apparatus according to clause 3, wherein identifying the first depth data and the second depth data from the depth data of the field of view comprises fitting a subsurface scattering model to individual peaks of the multi-peak point cloud by adapting parameters of the subsurface scattering model and identifying, by the trained model, whether each peak is associated with an object, a reflection of an object or a reflective surface based on the adapted parameters of the subsurface scattering model.

[0291] 5) The apparatus according to any preceding clause, wherein the trained model is trained on training data comprising depth data of objects of a known type within the field of view of the depth sensor.

[0292] 6) The apparatus according to any preceding clause, wherein determining the location data for the object for which the reflection of the object is received based on the first depth data in relation to the second depth data comprises correcting the first depth data based on the second depth data.

[0293] 7) The apparatus according to any preceding clause, wherein correcting the first depth data comprises performing plane fitting to identify a plane of the reflective surface from the second depth data and transforming the first depth data about the plane of the reflective surface.

[0294] 8) The apparatus according to clause 7, wherein transforming the first depth data about the plane of the reflective surface comprises flipping the first depth data about the plane of the reflective surface.9) The apparatus according to any preceding clause, wherein the circuitry is further configured to remove at least one of the first or second depth data from the depth data of the field of view. 10) The apparatus according to any preceding clause, wherein the circuitry is further configured to determine, from the depth data, third depth data representing a second object, wherein the depth detection circuitry, the reflective surface and the second object are located in that order. 11) The apparatus according to any preceding clause, wherein the trained model is a machine learning model.

[0295] 12) A SLAM, autonomous navigation or 3D scanning system comprising the apparatus according to any preceding clause.

[0296] 13) A method of determining location data for an object, the method comprising:

[0297] acquiring, from depth detection circuitry, depth data of a field of view of the depth detection circuitry;

[0298] identifying from the depth data of the field of view, by a trained model, first depth data representing depth data of a reflection of an object and second depth data representing depth data of a reflective surface, the reflective surface being the surface from which the reflection of the object is received; and

[0299] determining location data for the object for which the reflection of the object is received based on the first depth data in relation to the second depth data, the location data being indicative of a location of the object outside the field of view.

[0300] 14) A computer program comprising instructions which, when implemented by a computer, causes the computer to perform a method of determining location data for an object, the method comprising:

[0301] acquiring, from the depth detection circuitry, depth data of a field of view of the depth detection circuitry;

[0302] identifying from the depth data of the field of view, by a trained model, first depth data representing depth data of a reflection of an object and second depth data representing depth data of a reflective surface, the reflective surface being the surface from which the reflection of the object is received; and

[0303] determining location data for the object for which the reflection of the object is received based on the first depth data in relation to the second depth data, wherein the location data being indicative of a location outside a field of view.

[0304] 15) A non-transitory computer readable storage medium storing the computer program according to clause 14.

[0305] 16) An apparatus for controlling focusing, the apparatus comprising circuitry configured to: receive from a target field of view, via depth detection circuitry, data indicative of depth measurements for both a first object in the field of view, through an intermediate surface having reflective properties, and a depth measurement less than that of the first object and at least one reflection of an object;

[0306] generate respective data indicative of the depth measurements;identify from the respective data indicative of the depth measurements, by a trained model, first depth data representing the first object in the field of view and second depth data representing the reflection of the object; and

[0307] generate control signals to control focusing circuitry according to an input signal used to select one of the first depth data and the second depth data; wherein the circuitry is configured to generate the control signals to control the focusing circuitry based on a selected one of the first depth data and the second depth data.

[0308] 17) The apparatus according to clause 16, wherein the depth detection circuitry is a direct time-of-flight based depth detection circuitry.

[0309] 18) The apparatus according to clause 16 or 17, wherein the data indicative of depth measurements received from the target field of view is a multi-peak point cloud.

[0310] 19) The apparatus according to clause 18, wherein identifying the first depth data and the second depth data from the respective data indicate of the depth measurements comprises fitting a subsurface scattering model to individual peaks of the multi-peak point cloud by adapting parameters of the subsurface scattering model and identifying, by the trained model, whether each peak is associated with an object, a reflection of an object or a reflective surface based on the adapted parameters of the subsurface scattering model.

[0311] 20) The apparatus according to clause 18, wherein the trained model is trained on training data comprising depth data of objects of a known type within the field of view of the depth sensor.

[0312] 21) The apparatus according to any of clauses 16 to 20, wherein the circuitry is configured to receive the input signal via a user interface or a selection input device.

[0313] 22) The apparatus according to any of clauses 16 to 20, wherein the circuitry is configured to perform filtering to remove the other of the selected one of the first depth data and the second depth data before generating the control signals to control the focusing circuitry.

[0314] 23) The apparatus according to clause 22, wherein the circuitry is further configured to perform in-painting on the data indicative of depth measurements when removing the other of the selected one of the first depth data and the second depth data before generating the control signals to control the focusing circuitry.

[0315] 24) The apparatus according to any of clauses 16 to 23, wherein the trained model is a machine learning model.

[0316] 25) A method of controlling focusing, the method comprising:

[0317] receiving from a target field of view, via depth detection circuitry, data indicative of depth measurements for both a first object in the field of view, through an intermediate surface having reflective properties, and a depth measurement less than that of the first object and at least one reflection of an object;

[0318] generating respective data indicative of the depth measurements;

[0319] identifying from the respective data indicative of the depth measurements, by a trained model, first depth data representing the first object in the field of view and second depth data representing the reflection of the object; and

[0320] generating control signals to control focusing circuitry according to an input signal used to select one of the first depth data and the second depth data; wherein the circuitry is configured to generate thecontrol signals to control the focusing circuitry based on a selected one of the first depth data and the second depth data.

[0321] 26) A computer program comprising instructions which, when implemented by a computer, causes the computer to perform a method of controlling focusing, the method comprising:

[0322] receiving from a target field of view, via depth detection circuitry, data indicative of depth measurements for both a first object in the field of view, through an intermediate surface having reflective properties, and a depth measurement less than that of the first object and at least one reflection of an object;

[0323] generating respective data indicative of the depth measurements;

[0324] identifying from the respective data indicative of the depth measurements, by a trained model, first depth data representing the first object in the field of view and second depth data representing the reflection of the object; and

[0325] generating control signals to control focusing circuitry according to an input signal used to select one of the first depth data and the second depth data; wherein the circuitry is configured to generate the control signals to control the focusing circuitry based on a selected one of the first depth data and the second depth data.

[0326] 27) A non-transitory computer readable storage medium storing the computer program according to clause 26.

[0327] 28) An apparatus for controlling focusing, the apparatus comprising circuitry configured to: receive from a target field of view, via depth detection circuitry, data indicative of depth measurements for both a first object in the field of view, through an intermediate surface having reflective properties, and a depth measurement less than that of the first object and at least one reflection of an object;

[0328] generate respective data indicative of the depth measurements;

[0329] identify from the respective data indicative of the depth measurements, by a trained model, first depth data representing the first object in the field of view and second depth data representing the reflection of the object;

[0330] perform phase detection auto focus to determine a temporary focus point in the field of view; select one of the first depth data and the second depth data based on a comparison of the first depth data and second depth data with the temporary focus point from the phase detection auto focus; and

[0331] generate control signals to control focusing circuitry according to the selected one of the first depth data and the second depth data.

[0332] 29) The apparatus according to clause 28, wherein the depth detection circuitry is a direct time-of-flight base depth detection circuitry.

[0333] 30) The apparatus according to clause 28 or 29, wherein the data indicative of depth measurements received from the target field of view is a multi-peak point cloud.

[0334] 31) The apparatus according to clause 30, wherein identifying the first depth data and the second depth data from the respective data indicate of the depth measurements comprises fitting a subsurface scattering model to individual peaks of the multi-peak point cloud by adapting parameters of the subsurface scattering model and identifying, by the trained model, whether eachpeak is associated with an object, a reflection of an object or a reflective surface based on the adapted parameters of the subsurface scattering model.

[0335] 32) The apparatus according to clause 31, wherein the trained model is trained on training data comprising depth data of objects of a known type within the field of view of the depth sensor.

[0336] 33) The apparatus according to any of clauses 28 to 32, wherein the circuitry is configured to perform filtering to remove the other of the selected one of the first depth data and the second depth data before generating the control signals to control the focusing circuitry.

[0337] 34) The apparatus according to clause 33, wherein the circuitry is further configured to perform in-painting on the data indicative of depth measurements when removing the other of the selected one of the first depth data and the second depth data before generating the control signals to control the focusing circuitry.

[0338] 35) The apparatus according to any of clauses 28 to 34, wherein the trained model is a machine learning model.

[0339] 36) A method of controlling focusing, the method comprising:

[0340] receiving from a target field of view, via depth detection circuitry, data indicative of depth measurements for both a first object in the field of view, through an intermediate surface having reflective properties, and a depth measurement less than that of the first object and at least one reflection of an object;

[0341] generating respective data indicative of the depth measurements;

[0342] identifying from the respective data indicative of the depth measurements, by a trained model, first depth data representing the first object in the field of view and second depth data representing the reflection of the object;

[0343] performing phase detection auto focus to determine a temporary focus point in the field of view;

[0344] selecting one of the first depth data and the second depth data based on a comparison of the first depth data and second depth data with the temporary focus point from the phase detection auto focus; and

[0345] generating control signals to control focusing circuitry according to the selected one of the first depth data and the second depth data.

[0346] 37) A computer program comprising instructions which, when implemented on a computer, cause the computer to perform a method of controlling focusing, the method comprising:

[0347] receiving from a target field of view, via depth detection circuitry, data indicative of depth measurements for both a first object in the field of view, through an intermediate surface having reflective properties, and a depth measurement less than that of the first object and at least one reflection of an object;

[0348] generating respective data indicative of the depth measurements;

[0349] identifying from the respective data indicative of the depth measurements, by a trained model, first depth data representing the first object in the field of view and second depth data representing the reflection of the object;

[0350] performing phase detection auto focus to determine a temporary focus point in the field of view;selecting one of the first depth data and the second depth data based on a comparison of the first depth data and second depth data with the temporary focus point from the phase detection auto focus; and

[0351] generating control signals to control focusing circuitry according to the selected one of the first depth data and the second depth data.

[0352] 38) A non-transitory computer readable storage medium storing the computer program according to clause 37.

[0353] In so far as embodiments of the disclosure have been described as being implemented, at least in part, by software-controlled data processing apparatus, it will be appreciated that a non-transitory machine-readable medium carrying such software, such as an optical disk, a magnetic disk, semiconductor memory or the like, is also considered to represent an embodiment of the present disclosure.

[0354] It will be appreciated that the above description for clarity has described embodiments with reference to different functional units, circuitry and / or processors. However, it will be apparent that any suitable distribution of functionality between different functional units, circuitry and / or processors may be used without detracting from the embodiments.

[0355] Described embodiments may be implemented in any suitable form including hardware, software, firmware or any combination of these. Described embodiments may optionally be implemented at least partly as computer software running on one or more data processors and / or digital signal processors. The elements and components of any embodiment may be physically, functionally and logically implemented in any suitable way. Indeed the functionality may be implemented in a single unit, in a plurality of units or as part of other functional units. As such, the disclosed embodiments may be implemented in a single unit or may be physically and functionally distributed between different units, circuitry and / or processors.

[0356] Although the present disclosure has been described in connection with some embodiments, it is not intended to be limited to the specific form set forth herein. Additionally, although a feature may appear to be described in connection with particular embodiments, one skilled in the art would recognize that various features of the described embodiments may be combined in any manner suitable to implement the technique. Indeed, features which have been described with reference to one of the example apparatus 3000, example apparatus 12000 and example apparatus 17000 may be combined with features of the others of example apparatus 3000, example apparatus 12000 and example apparatus 17000 unless such combination is explicitly excluded.

Claims

CLAIMS1. An apparatus for determining location data for an object, the apparatus comprising circuitry configured to:acquire, from depth detection circuitry, depth data of a field of view of the depth detection circuitry;identify from the depth data of the field of view, by a trained model, first depth data representing depth data of a reflection of an object and second depth data representing depth data of a reflective surface, the reflective surface being the surface from which the reflection of the object is received; anddetermine location data for the object for which the reflection of the object is received based on the first depth data in relation to the second depth data, wherein the location data being indicative of a location outside a field of view.

2. The apparatus according to claim 1, wherein the depth detection circuitry is a direct time-of-flight based sensor.

3. The apparatus according to claim 1, wherein the depth data comprises a multi-peak point cloud.

4. The apparatus according to claim 3, wherein identifying the first depth data and the second depth data from the depth data of the field of view comprises fitting a subsurface scattering model to individual peaks of the multi-peak point cloud by adapting parameters of the subsurface scattering model and identifying, by the trained model, whether each peak is associated with an object, a reflection of an object or a reflective surface based on the adapted parameters of the subsurface scattering model.

5. The apparatus according to claim 1, wherein the trained model is trained on training data comprising depth data of objects of a known type within the field of view of the depth sensor.

6. The apparatus according to claim 1, wherein determining the location data for the object for which the reflection of the object is received based on the first depth data in relation to the second depth data comprises correcting the first depth data based on the second depth data.

7. The apparatus according to claim 1, wherein correcting the first depth data comprises performing plane fitting to identify a plane of the reflective surface from the second depth data and transforming the first depth data about the plane of the reflective surface.

8. The apparatus according to claim 7, wherein transforming the first depth data about the plane of the reflective surface comprises flipping the first depth data about the plane of the reflective surface.

9. The apparatus according to claim 1, wherein the circuitry is further configured to remove at least one of the first or second depth data from the depth data of the field of view.

10. The apparatus according to claim 9, wherein the circuitry is further configured to determine, from the depth data, third depth data representing a second object, wherein the depth detection circuitry, the reflective surface and the second object are located in that order.

11. The apparatus according to claim 1, wherein the trained model is a machine learning model.

12. A SLAM, autonomous navigation or 3D scanning system comprising the apparatus according to claim 1.

13. A method of determining location data for an object, the method comprising:acquiring, from depth detection circuitry, depth data of a field of view of the depth detection circuitry;identifying from the depth data of the field of view, by a trained model, first depth data representing depth data of a reflection of an object and second depth data representing depth data of a reflective surface, the reflective surface being the surface from which the reflection of the object is received; anddetermining location data for the object for which the reflection of the object is received based on the first depth data in relation to the second depth data, the location data being indicative of a location of the object outside the field of view.

14. A computer program comprising instructions which, when implemented by a computer, causes the computer to perform a method of determining location data for an object, the method comprising:acquiring, from depth detection circuitry , depth data of a field of view of the depth detection circuitry;identifying from the depth data of the field of view, by a trained model, first depth data representing depth data of a reflection of an object and second depth data representing depth data of a reflective surface, the reflective surface being the surface from which the reflection of the object is received; anddetermining location data for the object for which the reflection of the object is received based on the first depth data in relation to the second depth data, wherein the location data being indicative of a location outside a field of view.

15. A non-transitory computer readable storage medium storing the computer program according to claim 14.

Images