Electronic device, method and computer program

Abstract

An electronic device, comprising circuitry configured to determine Time-of-Flight histogram data from raw data obtained by a Time-of-Flight pixel array; generate, based on spot coordinates of spots received by the Time-of-Flight pixel array, a dot mask, the dot mask being indicative of dot pixels; select, based on the dot mask, one or more dot pixels; select, based on the dot mask and based on a predetermined set of coordinates on the Time-of Flight pixel array, one or more valley pixels, and calculate one or more residual histograms by subtracting one or more valley histograms, generated by the valley pixels, from one or more dot histograms, generated by the dot pixels; and generate depth information based on the one or more residual histograms.

Description

[0001] Sony Semiconductor Solutions Corporation et al.

[0002] ELECTRONIC DEVICE, METHOD AND COMPUTER PROGRAM

[0003] TECHNICAL FIELD

[0004] The present disclosure generally pertains to time-of-flight devices, time-of flight methods and corresponding computer programs.

[0005] TECHNICAL BACKGROUND

[0006] Time-of-Flight (ToF) technology is a widely used method for depth sensing and distance measurement in various applications, including 3D imaging, autonomous vehicles, robotics, and augmented reality. ToF sensors operate by emitting light pulses towards a scene and measuring the time it takes for the light to reflect back from objects in the scene to the sensor. This time delay, known as the time-of-flight, is directly proportional to the distance between the sensor and the object, allowing for the creation of depth maps and 3D representations of the environment.

[0007] Direct Time-of-Flight (dToF) involves measuring the time it takes for a light pulse to travel from the emitter to the object and back to the receiver. This method provides high accuracy and is particularly effective for long-range measurements. In dToF systems, a light source, such as a Vertical-Cavity Surface-Emitting Laser (VCSEL) array, emits short light pulses towards the scene. The reflected light is then detected by a ToF receiver array. The time delay measurement involves collecting multiple returning photons, and the results are compiled into a histogram. This histogram records the number of detected photons over discrete time bins, effectively representing the time distribution of the returned light. The accuracy of dToF systems depends on the precision of the time delay measurement, which is typically in the order of nanoseconds or picoseconds.

[0008] A significant challenge in ToF technology is the presence of noise and disturbances, such as sunlight interference, multipath reflections, and cover glass noise, which can degrade the accuracy and reliability of the measurements. To address these challenges, advanced signal processing techniques are employed to identify the wanted signal in the detected photons and / or to separate the wanted signal from unwanted noise.

[0009] One such technique is Direct and Global Separation (DGS). The DGS algorithm is designed to separate the direct component of the signal, which represents the actual distance measurement, from the global component, which includes various disturbances and noise. The DGS algorithm processes the histograms generated by the ToF sensor to isolate the direct signal. The DGS algorithm operates by leveraging the spatial and temporal characteristics of the ToF data. Sony Semiconductor Solutions Corporation et al.

[0010] In order to improve the accuracy of ToF measurements, an improved DGS algorithm is desirable.

[0011] SUMMARY

[0012] According to a first aspect the disclosure provides an electronic device according to independent claim 1. According to a second aspect, the disclosure provides a method according to independent claim 25. According to a third aspect, the disclosure provides a method according to independent claim 26. Further aspects are set forth in the dependent claims, the drawings and the following description.

[0013] BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Embodiments are explained by way of example with respect to the accompanying drawings, in which:

[0015] Fig. 1 shows a Time-of-Flight (ToF) sensor including a VCSEL array and a ToF receiver array for depth measurement; and

[0016] Fig. 2 shows a method for processing ToF raw data by a dToF system and a software datapath including an improved DGS algorithm to produce depth and SNR outputs; and

[0017] Fig. 3 shows a method for processing spot coordinates and raw histograms within a DGS algorithm using dot and valley histogram selection, noise floor removal, and offset subtraction / addition; and

[0018] Fig. 4A shows a more detailed explanation of the dot mask creation step from spot coordinates; and

[0019] Fig. 4B shows an illustration of a dot mask overlayed onto a sensor array; and

[0020] Fig. 5 shows a detailed illustration of the dot histogram selection step, which selects dot histograms for further processing; and

[0021] Fig. 6A shows a detail of a sensor array with an example dot distribution and a dot binning window encompassing multiple dot pixels; and

[0022] Fig. 6B shows a detail of a sensor array with an example dot distribution and a dot binning window encompassing a single dot pixel; and

[0023] Fig. 7A shows nine dot histograms overlaid on each other, as may be output by multiple dot pixels; and

[0024] Fig. 7B shows an example of a single dot histogram; and Sony Semiconductor Solutions Corporation et al.

[0025] Fig. 8 shows the valley histogram selection step, which selects valley histograms for further processing; and

[0026] Fig. 9A shows a detail of a sensor array with an example distribution of four valley pixels arranged in a circle around a dot pixel; and

[0027] Fig. 9B shows a detail of a sensor array with an example distribution of six valley pixels arranged in a circle around a dot pixel; and

[0028] Fig. 9C shows a detail of a sensor array with an example distribution of six valley pixels arranged in a larger circle around a dot pixel; and

[0029] Fig. 9D shows a detail of a sensor array with an example distribution of eight valley pixels arranged in a circle around a dot pixel; and

[0030] Fig. 10 shows the histogram filtering step, which orders and selects dot histograms based on their distance to the histogram at the spot coordinate; and

[0031] Fig. 11 A shows an illustration of a plurality of dot histograms resulting from dots subtending a selection window; and

[0032] Fig. 1 IB shows an illustration of the filtering of dot histograms, resulting in fewer selected histograms; and

[0033] Fig. 12A shows a detailed explanation of the spatial average calculation step for dot histograms; and

[0034] Fig. 12B shows a detailed explanation of the spatial average calculation step for valley histograms; and

[0035] Fig. 13 A shows an illustrative input of multiple histograms for spatial averaging; and

[0036] Fig. 13B shows an averaged histogram resulting from the spatial averaging of multiple histograms; and

[0037] Fig. 14A shows a detailed explanation of the noise floor removal step for dot histograms; and

[0038] Fig. 14B shows a detailed explanation of the noise floor removal step for valley histograms; and

[0039] Fig. 15A shows an illustration of histograms to which the mean noise floor removal is applied; and

[0040] Fig. 15B shows an illustration of histograms after the mean noise floor removal is applied; and

[0041] Fig. 16A shows an illustration of histograms to which the median noise floor removal is applied; and Sony Semiconductor Solutions Corporation et al.

[0042] Fig. 16B shows an illustration of histograms after the median noise floor removal is applied; and

[0043] Fig. 17 shows the scaling step, which scales the valley histograms with respect to the dot histograms; and

[0044] Fig. 18 illustrates the scaling of a histogram by a factor of 2; and

[0045] Fig. 19A illustrates the offset subtraction step for dot histograms; and

[0046] Fig. 19B illustrates the offset addition step for valley histograms; and

[0047] Fig. 20 illustrates offsetting a histogram by subtracting an offset value; and

[0048] Fig. 21 describes the histogram subtraction step, which calculates residual histograms by subtracting valley histograms from dot histograms; and

[0049] Fig. 22A illustrates the input histograms for the histogram subtraction step, showing dot and valley histograms with strong interference; and

[0050] Fig. 22B illustrates the output residual histogram after subtracting valley histograms from dot histograms; and

[0051] Fig. 23 A illustrates the input histograms for the histogram subtraction step, showing dot and valley histograms with weak interference; and

[0052] Fig. 23B illustrates the output residual histogram after subtracting valley histograms from dot histograms; and

[0053] Fig. 24A shows an input histogram for the histogram subtraction step with a noise floor corresponding to strong sunlight; and

[0054] Fig. 24B illustrates the output residual histogram after subtracting valley histograms from dot histograms with a noise floor corresponding to strong sunlight; and

[0055] Fig. 25A shows an input histogram for the histogram subtraction step with a noise floor corresponding to strong sunlight and median noise floor processing; and

[0056] Fig. 25B illustrates the output residual histogram after subtracting valley histograms from dot histograms with median noise floor processing; and

[0057] Fig. 26 shows the depth and SNR calculation step, which uses residual histograms to calculate depth and SNR values; and

[0058] Fig. 27A shows a ToF receiver array with depth values indicated for a high SNR threshold; and

[0059] Fig. 27B shows a ToF receiver array with depth values indicated for a low SNR threshold; and Sony Semiconductor Solutions Corporation et al.

[0060] Fig. 28A shows an alternative dynamic selection of dots based on an SNR noise metric; and

[0061] Fig. 28B shows an alternative dynamic selection of valleys based on an SNR noise metric; and

[0062] Fig. 29 illustrates the dynamic selection of dots and valleys based on decreasing SNR; and

[0063] Fig. 30 shows a circuitry comprising various components including a CPU, image sensor, ToF interface, and other communication interfaces.

[0064] DETAILED DESCRIPTION OF EMBODIMENTS

[0065] Before a detailed description of the embodiments under reference of Fig. 1 is given, general explanations are made.

[0066] An electronic device, comprises circuitry configured to determine Time-of-Flight histogram data from raw data obtained by a Time-of-Flight pixel array; generate, based on spot coordinates of spots received by the Time-of-Flight pixel array, a dot mask, the dot mask being indicative of dot pixels; select, based on the dot mask, one or more dot pixels; select, based on the dot mask and based on a predetermined set of coordinates on the Time-of Flight pixel array, one or more valley pixels, and calculate one or more residual histograms by subtracting one or more valley histograms, generated by the valley pixels, from one or more dot histograms, generated by the dot pixels; and generate depth information based on the one or more residual histograms.

[0067] The electronic device may be a camera, smartphone, or any other device capable of capturing depth data and determining events from the captured data. It comprises circuitry configured to determine Time-of-Flight histogram data from raw data obtained by a Time-of-Flight pixel array. The Time-of-Flight pixel array may be a sensor array that detects the time it takes for emitted light to reflect back from objects, allowing for depth measurement. The Time-of-Flight pixel array may comprise single-photon-avalanche diodes (SPADs). The circuitry may include processors and control units that manage the data processing tasks. An example of circuitry may be the control circuitry in a smartphone that processes image data.

[0068] The spot coordinates may refer to specific locations on the sensor array that have been predetermined during calibration. The dot mask may be a matrix or array that indicates the positions of dot pixels, which are the pixels expected to receive the direct signal from the emitted light pulses.

[0069] Dot pixels may be the pixels within the dot mask that are expected to contain the wanted (direct) signal. The selection process may involve identifying these pixels based on their coordinates within the dot mask. The dot pixels are not required to correspond to an area of highest light intensity on the Time-of-Flight pixel array or to a local maximum of the light intensity in the Sony Semiconductor Solutions Corporation et al.

[0070] Time-of-Flight pixel array. However, there may be embodiments where the dot pixels correspond to an area of highest light intensity on the Time-of-Flight pixel array or to a local maximum of the light intensity in the Time-of-Flight pixel array

[0071] In some embodiments the dot mask may be generated by searching for the local maximum in received light intensity corresponding to each emitted spot in each frame across all pixels, correlated with the location of the emitter of the spot in the emitter array, in order to generate the dot mask. Each found maximum may be designated a dot and the pixel located at the dot may be designated a dot pixel.

[0072] In other embodiments, the dot mask may be generated by searching, based on the known spot coordinates, for local maxima of light intensity in a small area around each known spot coordinate. The small area may, for example, comprise a circle with a radius corresponding to half the distance to the next known spot coordinate. A maximum found in the small area around a spot may be designated a dot corresponding to the spot and the pixel located at the dot may be designated a dot pixel.

[0073] In yet other embodiments, the dot mask may be generated by designating the pixel located at the spot coordinate as a dot pixel.

[0074] Valley pixels may be pixels located in regions between the dots, where the light is not directly illuminated by the primary light source. The valley pixels may be selected based on the predetermined set of coordinates on the Time-of Flight pixel array. The predetermined set of coordinates may indicate one or more locations of pixels relative to a dot pixel. In some embodiments, a single pixel is indicated as a valley pixel. In other embodiments more than one pixel is indicated as valley pixels. If more than one valley pixel is indicated, the predetermined set of coordinates may indicate valley pixels arranged in a predetermined geometrical pattern. The predetermined geometrical pattern may be a circle or any other shape that defines the arrangement of valley pixels around the dot pixels. The valley pixels are not required to correspond to an area of lowest light intensity on the Time-of-Flight pixel array or to a local minimum of the light intensity in the Time-of-Flight pixel array.

[0075] In the following, a dot pixel may be referred to as “dot” and a valley pixel may be referred to as “valley”. Correspondingly, a plurality of dot pixels may be referred to as “dots” and a plurality of valley pixels may be referred to as “valleys”.

[0076] Subtracting, leading to the residual histogram, may refer to the process of removing the values of the valley histograms from the values of the dot histograms. The subtracting may be preformed Sony Semiconductor Solutions Corporation et al. on a bin-by-bin basis. The subtracting may be provided such that negative values resulting from the subtracting are set (clipped) to zero.

[0077] Residual histograms may be the result of this subtraction, isolating the direct component of the signal from the global component, which includes noise and disturbances.

[0078] Depth information may be the calculated distances between the sensor and the objects in the scene, derived from the time-of-flight data processed through the residual histograms. This depth information may be used for various applications, such as 3D imaging, augmented reality, and autonomous navigation.

[0079] There are embodiments where the circuitry is further configured to select the one or more valley pixels such that the selected valley pixels are arranged in the predetermined geometrical pattern on the Time-of-Flight pixel array. The Time-of-Flight pixel array includes a plurality of pixels arranged in a two-dimensional pattern. Selecting the valley pixels such that the selected valley pixels are arranged in the predetermined geometrical pattern on the Time-of-Flight pixel array means selecting pixels on the Time-of-Flight pixel array such that the positions of the selected pixels correspond to points of the predetermined geometrical pattern.

[0080] The predetermined geometrical pattern may be a specific arrangement or shape used to organize the valley pixels around the dot pixels. The geometrical pattern may be a circle, square, or any other shape that defines the spatial relationship between the valley pixels. An example of a predetermined geometrical pattern may be a circular arrangement of valley pixels around a position on the Time-of-Flight pixel array. The predetermined geometrical pattern may be included in setting data or calibration data of the device, or may be predetermined based on parameters determined by the device or by user setting.

[0081] There are embodiments where the plurality of selectable geometrical patterns are selectable based on pattern parameters. Selectable geometrical patterns may be specific arrangements or shapes that can be chosen to organize pixels, such as valley pixels, on a sensor array. These patterns may include circles, squares, or other geometric shapes that define the spatial relationship between pixels. An example of selectable geometrical patterns may be different circular arrangements of valley pixels around a central dot pixel on a Time-of-Flight sensor array.

[0082] Pattern parameters may be the criteria or variables used to define and select the geometrical patterns. These parameters may include dimensions, radii, or other characteristics that determine the shape and arrangement of the patterns. An example of pattern parameters may be the radius Sony Semiconductor Solutions Corporation et al. of a circle or the number of pixels to be included in a specific geometric arrangement on a sensor array.

[0083] There are embodiments where the circuitry is further configured to select the predetermined geometrical pattern based on a noise metric. A noise metric may be a measure or evaluation of the amount of noise present in the signal captured by the sensor array. This metric may be used to assess the quality of the signal and to make decisions about processing steps, such as selecting geometrical patterns. An example of a noise metric may be the signal-to-noise ratio (SNR) calculated from the time-of-flight data to determine the level of interference or disturbances in the measurements. The SNR may be calculated based on the dot histograms or the valley histograms, or may be provided by another component of the device.

[0084] There are embodiments where the predetermined geometrical pattern is a circle centered on one of the spot coordinates. Spot coordinates may refer to specific locations on the sensor array that have been predetermined during calibration. These coordinates identify the pixels that are expected to receive the direct signal from the emitted light pulses, as well as any potential disturbances or noise. An example of spot coordinates may be the specific pixel locations on a Time-of-Flight sensor array that are used to generate a dot mask for selecting dot and valley pixels.

[0085] There are embodiments where the predetermined geometrical pattern is a circle with a predetermined radius. A predetermined radius may refer to the fixed distance from the center of the circle to any point on its circumference. This radius may be defined based on the specific requirements of the sensor array and the desired arrangement of the valley pixels. An example of a predetermined radius may be the distance set during the calibration of a Time-of-Flight sensor array to ensure that valley pixels are appropriately spaced around the dot pixels to effectively capture noise and disturbances. The predetermined radius may be included in setting data or calibration data of the device, or may be predetermined based on parameters determined by the device or by user setting.

[0086] There are embodiments where the one or more valley pixels are a predetermined number of valley pixels arranged regularly along the circle.

[0087] Arranged regularly may refer to the even and consistent placement of valley pixels along a defined path or shape, such as a circle. This regular arrangement may ensure that the valley pixels are evenly spaced to provide uniform coverage for capturing the global component of the signal. An example of arranged regularly may be the placement of valley pixels at equal intervals Sony Semiconductor Solutions Corporation et al. along the circumference of a circle centered on a dot pixel, a spot coordinate or any position on the Time-of-Flight pixel array.

[0088] The predetermined number of valley pixels may be included in setting data or calibration data of the electronic device, or may be predetermined based on parameters determined by the device or by user setting.

[0089] There are embodiments where the circuitry is further configured to select the predetermined number of valley pixels based on a noise metric.

[0090] The predetermined number may refer to a specific quantity that has been set in advance based on the requirements of the sensor array and the desired arrangement of the valley pixels. This number may be determined during the calibration or configuration of the device, or may be predetermined based on a mapping of the noise metric on a number representing the number of valley pixels. An example of a predetermined number may be the number of valley pixels selected to be placed around a dot pixel, a spot coordinate or any position on the Time-of-Flight pixel array.

[0091] There are embodiments where the circuitry is further configured to determine a noise floor of the one or more valley histograms and subtract the noise floor of the one or more valley histograms from the one or more valley histograms prior to calculating the one or more residual histograms. The subtracting may be provided on a bin-by-bin basis. The subtracting may be provided such that negative values resulting from the subtracting are set (clipped) to zero.

[0092] The noise floor may be the level of background noise present in the signal captured by the sensor array. This noise floor may be determined by analyzing the signal to identify the baseline level of noise that is consistently present. An example of a noise floor may be the average level of ambient light detected by a sensor array in a Time-of-Flight camera. The noise floor may be determined from the valley histogram by calculating a global, local, running or segmented mean of the valley histogram, or by calculating a global, local, running or segmented median of the valley histogram.

[0093] Subtracting the noise floor may refer to the process of removing the baseline level of noise from the valley histograms on a bin-by-bin basis. An example of subtracting the noise floor may be adjusting the values in a histogram to account for the average level of ambient light detected by the sensor array.

[0094] There are embodiments where the noise floor is a running mean of the valley histogram or a running median of the valley histogram. Sony Semiconductor Solutions Corporation et al.

[0095] A running mean may refer to the average value calculated over a moving window of data points within the valley histogram. This method may smooth out short-term fluctuations and highlight longer-term trends in the noise level. An example of a running mean may be the average of the last ten data points in a histogram, recalculated as new data points are added.

[0096] A running median may refer to the middle value in a moving window of data points within the valley histogram. This method may reduce the impact of outliers and provide a more robust measure of the central tendency of the noise level. An example of a running median may be the median of the last ten data points in a histogram, recalculated as new data points are added.

[0097] There are embodiments where the circuitry is further configured to spatially average the one or more valley histograms prior to calculating the one or more residual histograms.

[0098] Spatially averaging may refer to the process of combining data from multiple spatial locations to produce a single averaged value. This method may reduce noise and improve the signal quality by averaging the values of corresponding bins across multiple histograms within a defined spatial region. An example of spatially averaging may be calculating the average value of time- of-flight measurements from several adjacent or discontinuous valley pixels on a sensor array.

[0099] There are embodiments where the circuitry is further configured to add a predetermined valley histogram offset from the one or more valley histograms prior to calculating the one or more residual histograms.

[0100] A predetermined valley histogram offset may refer to a specific value that has been set in advance to adjust the valley histograms. The predetermined valley histogram offset may be determined based on the characteristics of the sensor array and the expected level of background noise. An example of a adding a predetermined valley histogram offset may be a fixed number of counts added to each bin in the histogram. The predetermined valley histogram offset may be included in setting data or calibration data of the device, or may be predetermined based on parameters determined by the device or by user setting.

[0101] There are embodiments where the circuitry is further configured to select the one or more dot pixels based on whether a dot pixel is inside or outside of a selection window.

[0102] The selection window may refer to a defined area or region within the sensor array used to determine which pixels are considered for processing. This window may be based on specific criteria, such as the location of the spot coordinates or the desired area of interest. An example of a selection window may be a circular or rectangular region centered on a dot pixel, within which the pixels are selected for further analysis. Sony Semiconductor Solutions Corporation et al.

[0103] Inside or outside may refer to the relative position of the dot pixels with respect to the boundaries of the selection window. Pixels inside the selection window may be included for processing, while those outside may be excluded. An example of inside or outside may be determining whether a pixel falls within the defined circular region around a spot coordinate on the sensor array.

[0104] There are embodiments where the selection window is a circle with a predetermined radius centered on one of the spot coordinates.

[0105] There are embodiments where the predetermined radius is selectable based on selection window parameters. Selection window parameters may refer to the criteria or variables used to define and adjust the selection window. These parameters may include dimensions, radii, or other characteristics that determine the shape and size of the selection window. An example of selection window parameters may be the radius and center coordinates used to define a circular selection window on a sensor array for capturing time-of-flight data. The predetermined radius may be included in setting data or calibration data of the device, or may be predetermined based on parameters determined by the device or by user setting.

[0106] There are embodiments where the circuitry is further configured to select the predetermined radius based on a noise metric.

[0107] The noise metric may be an SNR that is calculated based on the dot histograms or the valley histograms, or may be provided by another component of the device.

[0108] There are embodiments where the circuitry is further configured to select, prior to calculating the one or more residual histograms, from the one or more dot histograms, a predetermined number of dot histograms having a lower two-norm distance with respect to a histogram corresponding to a spot coordinate of the one of the spots received by the Time-of-Flight receiver array than all other dot histograms of the one or more dot histograms.

[0109] The two-norm distance may refer to a mathematical measure of the difference between two histograms, calculated as the Euclidean distance between their corresponding values. This distance may be used to compare the similarity between histograms. An example of a two-norm distance may be the calculation of the Euclidean distance between the values of two time-of- flight histograms to determine their similarity.

[0110] The histogram corresponding to a spot coordinate may refer to the specific histogram generated by the pixels at a particular spot coordinate on the sensor array. This histogram may represent the time-of-flight data for that specific location. An example of a histogram corresponding to a spot Sony Semiconductor Solutions Corporation et al. coordinate may be the time-of-flight histogram generated by the pixels at a predetermined location on a sensor array during calibration.

[0111] There are embodiments where the circuitry is further configured to select, prior to calculating the one or more residual histograms, from the one or more dot histograms, a predetermined number of dot histograms having a higher peak value than all other dot histograms of the one or more dot histograms.

[0112] The predetermined number may refer to a specific quantity that has been set in advance based on the requirements of the sensor array and the desired arrangement of the pixels. This number may be determined during the calibration or configuration of the device. An example of a predetermined number may be the number of dot histograms selected for further processing to ensure accurate depth measurement. The predetermined number may be included in setting data or calibration data of the device, or may be predetermined based on parameters determined by the device or by user setting.

[0113] The peak value may refer to the highest value in a histogram, representing the most frequent or significant measurement within the data set. In the context of dot histograms, the peak value may indicate the most likely time-of-flight measurement for a particular pixel. An example of the peak value may be the highest count in a time-of-flight histogram, corresponding to the most common distance measurement for a specific pixel on the sensor array.

[0114] There are embodiments where the circuitry is further configured to determine a noise floor of the dot histogram and subtract the noise floor of the dot histogram from the dot histogram prior to calculating the one or more residual histograms. The subtracting may be provided on a bin-by- bin basis. The subtracting may be provided such that negative values resulting from the subtracting are set (clipped) to zero.

[0115] There are embodiments where the noise floor is a running mean of the dot histogram or a running median of the dot histogram.

[0116] There are embodiments where the circuitry is further configured to spatially average the one or more dot histograms prior to calculating the one or more residual histograms.

[0117] There are embodiments where the circuitry is further configured to subtract a predetermined dot histogram offset from the one or more dot histograms prior to calculating the one or more residual histograms. The subtracting may be provided on a bin-by-bin basis. The subtracting may be provided such that negative values resulting from the subtracting are set (clipped) to zero. Sony Semiconductor Solutions Corporation et al.

[0118] The predetermined dot histogram offset may be different from the a predetermined valley histogram offset. The predetermined dot histogram offset may be included in setting data or calibration data of the device, or may be predetermined based on parameters determined by the device or by user setting.

[0119] There are embodiments where the circuitry is further configured to scale the one or more valley histograms by a predetermined scaling factor prior to calculating the one or more residual histogram.

[0120] Scaling may refer to the process of multiplying the values in the valley histograms by a specific factor to adjust their magnitude. This adjustment may be necessary to ensure that the noise component is accurately represented when subtracted from the dot histograms. An example of scaling may be increasing the values in a histogram by a factor of two to account for underestimation of the global signal.

[0121] The predetermined scaling factor may refer to a specific value that has been set in advance to adjust the valley histograms. This factor may be determined based on the characteristics of the sensor array and the expected level of background noise. An example of a predetermined scaling factor may be a fixed number, such as 1.5 or 2, used to multiply the values in the valley histograms to ensure accurate noise representation. The scaling factor may be included in setting data or calibration data of the device, or may be predetermined based on parameters determined by the device or by user setting.

[0122] A sensing method for a time-of-flight sensing device comprises determining Time-of-Flight histogram data from raw data obtained by a Time-of-Flight pixel array; generating, based on spot coordinates of spots received by the Time-of-Flight pixel array, a dot mask, the dot mask being indicative of dot pixels; selecting, based on the dot mask, one or more dot pixels; selecting, based on the dot mask and based on a predetermined geometrical pattern, one or more valley pixels, and calculating one or more residual histograms by subtracting one or more valley histograms, generated by the valley pixels, from one or more dot histograms, generated by the dot pixels; and generating depth information based on the one or more residual histograms.

[0123] The methods as described herein are also implemented in some embodiments as a computer program causing a computer and / or a processor to perform the method, when being carried out on the computer and / or processor. In some embodiments, also a non-transitory computer- readable recording medium is provided that stores therein a computer program product, which, when executed by a processor, such as the processor described above, causes the methods described herein to be performed. Sony Semiconductor Solutions Corporation et al.

[0124] Layout of a Time-of-Flight imaging device

[0125] Fig. 1 shows Time-of-Flight (ToF) sensor 1 in the sense of the present disclosure. The ToF sensor includes a VCSEL array 5 and a ToF receiver array 7. The VCSEL array 5 emits light light LI towards a scene (not shown). This emitted light LI travels to the objects within the scene and is reflected back as reflected light L2.

[0126] The ToF receiver array 7 is configured to detect the reflected light L2 and convert it into electrical signals. These signals are then processed to determine the time delay between the emission of the light LI and the reception of the reflected light L2, which is directly related to the distance of the objects from the camera.

[0127] The control circuitry 8 is integrated into the system to manage the operations of both the VCSEL array 5 and the ToF receiver array 7. The control circuitry 8 ensures synchronized emission and reception of light, processes the time delay information, and generates depth data for the scene.

[0128] The ToF sensor shown in Fig. 1 is, for example, a direct Time-of-Flight (dToF) sensor. A dToF sensor measures the distance by calculating the time it takes for emitted light to travel to the object and back to the sensor. In this method, a light source, such as the VCSEL array 5, emits a light pulse (LI) towards a scene. The light pulse reflects off the object and returns as reflected light (L2) to a ToF receiver array. The time delay between the emission and reception of the light pulse is measured by the control circuitry 8, and this time delay is directly proportional to the distance of the object from the sensor.

[0129] Indirect Time-of-Flight (iToF) is another method for ToF measurements. iToF, as opposed to dToF, measures the distance by analyzing the phase shift of modulated light rather than the direct travel time. In this method, the VCSEL array emits modulated light LI towards the object. The light reflects off the object and returns as reflected light L2 to the ToF receiver array.

[0130] Instead of measuring the time delay directly, the system measures the phase shift between the emitted and reflected light waves. This phase shift is then used to calculate the distance to the object.

[0131] In dToF, the measured time delay is recorded for multiple pulses over a period of time, resulting in a distribution of time-of-flight measurements. These measurements are then compiled into histograms by the control circuitry. These histograms, indicate the frequency of each time delay value. Generally, in a graphic depiction of the histogram, a horizontal axis (x-axis) of the histogram represents the time delay values, which correspond to the time it takes for the emitted light pulse LI to travel to the object and return as reflected light L2. These time delay values are typically measured in nanoseconds (ns) or picoseconds (ps), depending on the resolution and Sony Semiconductor Solutions Corporation et al. precision of the ToF system. Generally, in a graphic depiction of the histogram, a vertical axis (y-axis) of the histogram represents the count of occurrences for each time delay value. This axis indicates how many times a particular time delay was detected by the ToF receiver array over a series of emitted pulses. Higher counts of occurrence suggest that a specific time delay value is more common, indicating a more likely distance to the object. The peak of the histogram corresponds to the most likely time-of-flight value, which is used to calculate the distance to the object.

[0132] Direct and Global Separation (DGS) is a signal processing technique used in Time-of-Flight (ToF) systems to enhance the accuracy of depth measurements by isolating the direct signal from various sources of noise and disturbances. As shown in Fig. 1 above, in ToF technology, sensors emit light pulses towards a scene and measure the time it takes for the light to reflect back from objects, creating depth maps and 3D representations. However, this process can be affected by noise such as sunlight interference, multipath reflections, and other environmental factors. A DGS algorithm addresses these challenges by analyzing the histograms generated from the ToF data, separating the direct component of the signal, which represents the actual distance measurement, from the global component, which includes unwanted noise.

[0133] Integrated Time-of-Flight Data Processing with DGS for Enhanced Depth Accuracy

[0134] In the following embodiments are disclosed for processing direct Time-of-Flight (dToF) raw data to generate accurate depth and Signal-to-Noise Ratio (SNR) outputs using an improved DGS algorithm. The method involves two main processing entities: a dToF system and a software datapath. The dToF system performs data refinement through various preprocessing steps to clean and prepare the raw data. The software datapath includes the improved Direct and Global Separation (DGS) algorithm to further enhance the accuracy by separating the direct signal from noise and disturbances.

[0135] Fig. 2 shows a method for processing ToF raw data 200 through a dToF system 10 and a software datapath 11 including an improved DGS algorithm to produce depth and SNR outputs.

[0136] The dToF system 10 performs data refinement through various preprocessing steps to clean the raw data and process the raw data into histograms for further analysis. The dToF system outputs two main types of data. First, it produces the Flood output 110, which includes histograms that have undergone various preprocessing steps and depth data obtained from the processed histograms. Second, it outputs the histograms to the DGS step (SI 11) in the software datapath.

[0137] The software datapath 11 utilizes the histograms provided by the dToF system 10 and processes them with the Direct and Global Separation (DGS) algorithm Si l l to separate the direct Sony Semiconductor Solutions Corporation et al. component from any disturbance or global component. The software datapath processes the histograms provided by the dToF system 10 using the Direct and Global Separation (DGS) to produce primary outputs DGS Depth 105a and DGS SNR 105b as outputs.

[0138] In the software datapath 11, the histograms are processed using the Direct and Global Separation (DGS) algorithm in the DGS step Si l l. The DGS algorithm separates the direct component (wanted signal) from the global component (disturbance or noise).

[0139] Finally, the depth and SNR calculation step SI 12 calculates the depth and SNR from the processed histograms at the spot coordinates. The output of SI 12 is referred to as DGS Depth / DGS SNR 105, which includes Depth data and SNR data. These values represent the calculated depth and signal-to-noise ratio (SNR) at the spot coordinates.

[0140] Fig. 3 shows a method for processing spot coordinates 210 and histograms 220 within a DGS algorithm using dot and valley histogram selection, noise floor removal and offset subtract! on / additi on .

[0141] The method provides for selecting and post-processing of dot histograms (i.e. the direct component) and valley histograms (i.e. the global component). After both the dot histograms and the valley histograms have been preprocessed, the valley histograms are subtracted from the dot histograms in order to produce residual histograms (also called DGS histograms) which are then used in order to calculate depth information.

[0142] At step S21, a dot mask 211 is created based on spot coordinates 210 obtained in a calibration of the device, for example during manufacture. The spot coordinates 210 refer to the specific locations on the sensor array that have been predetermined during the calibration of the Time-of- Flight (ToF) camera. These coordinates identify the pixels that are expected to receive the direct signal from the emitted light pulses, as well as any potential disturbances or noise. The spot coordinates are stored in a calibration file and are known a priori, meaning they are established before the ToF system is used in its operational environment. The dot mask 211 identifies specific pixels on the sensor array that are expected to receive both the wanted (direct) signal and any unwanted disturbances (global signal). This is explained in more detail with regard to Fig. 4 and the corresponding description.

[0143] There are different methods to generate the dot mask based on the spot coordinates. According to one alternative, the dot mask is generated by designating a pixel located at a spot coordinate as a dot pixel. According to another alternative, a small area around the spot coordinate, but including the spot coordinate, is searched for a local maximum of incident light intensity and a pixel located at the maximum of incident light intensity as a dot pixel. Sony Semiconductor Solutions Corporation et al.

[0144] Both, the dot mask 211 obtained at S21 and the histograms 220 are then used by dot histogram selection S22 and valley histogram selection S28 to identify dot histograms and, respectively, valley histograms from histograms 220.

[0145] The histogram selection S22 selects dot pixels and designates the dot histograms, comprised in the histograms 220, generated by the selected dot pixels, for subsequent processing.

[0146] The histogram selection S28 selects valley pixels corresponding to a geometric pattern and based on the dot mask 211 and designates the valley histograms, comprised in the histograms 220, generated by the selected valley pixels, for subsequent processing.

[0147] After dot histogram selection S22, the method proceeds through a sequence of post-processing steps: histogram filtering S23, spatial average calculation S24, noise floor removal S25, and offset subtraction S26. The output of these is post-processed dot histograms 212, which contain the direct and the global components of the signal. At S23, histogram filtering is performed which orders the dot histograms based on their two-norm distance to the histogram at the spot coordinate and selects a specified number of dot histograms for further processing, discarding the rest, as described in more detail in Fig. 10 and the corresponding description. Then a spatial average calculation S24 averages the values of corresponding bins across multiple selected dot histograms to reduce noise and improve the signal quality. This step is described in more detail in Fig. 12A. At S25, a noise floor is removed from the histograms by calculating either the median or mean value in each segment of the histograms and subtracting it. This step is described in more detail in Fig. 14A. At S26, a predefined offset is subtracted from the histograms to reduce bias due to large noise floor values, particularly in strong sunlight scenes. This step is described in more detail in Fig. 19A.

[0148] After valley histogram selection S28, the method proceeds through a sequence of postprocessing steps: spatial average calculation step S29, noise floor removal step S210, scaling step S211, and offset addition step S212. The output of these steps is post-processed valley histograms 222, which primarily contain the global component of the signal. At S29, a spatial average calculation averages the values of corresponding bins across multiple selected valley histograms to reduce noise and improve the signal quality. This step is described in more detail in Fig. 12B. At S210, a noise floor is removed from the histograms by calculating either the median or mean value in each segment of the histograms and subtracting it. This step is described in more detail in Fig. 14B. At S211, the valley histograms are scaled with respect to the dot histograms to address situations where the global signal is underestimated, making the subtraction of the disturbance more effective. This step is described in more detail in Fig. 17. Sony Semiconductor Solutions Corporation et al.

[0149] Finally, at S212, a predefined offset is added to the valley histograms to reduce bias due to large noise floor values, particularly in strong sunlight scenes, as described in more detail in Fig. 19B.

[0150] Both the post-processed dot histograms 212 and the post-processed valley histograms 222 are then input into the histogram subtraction step S213. In this step, the valley histograms 222 are subtracted from the dot histograms 212 to isolate the direct component from the global component. Residual histograms resulting from the histogram subtraction step S213 contain only the direct component of the initial histograms 220. The residual histograms are marked DGS histograms in Fig. 3.

[0151] The residual histograms 230 are then passed from the DGS algorithm SI 11 to the depth and SNR calculation step SI 12 as described with reference to Fig. 2 above.

[0152] By the processing as described in the embodiment of Fig. 3, the raw histograms have been refined by isolating the direct signal (which represents the actual distance measurements) from the global signal (which includes various sources of noise and disturbances). The result is a set of DGS histograms 230 that provide a clearer and more accurate representation of the time-of-flight data.

[0153] Dot mask generation

[0154] Fig. 4A shows a more detailed explanation of the dot mask creation step S21 as shown in Fig. 3.

[0155] The dot mask creation step S21 involves creating a dot mask 211 from spot coordinates 210. Individual spot coordinates 30, as are comprised in the spot coordinates 210, are indicated in Fig. 4A. Spot coordinates correspond to one or more dot pixels 51. The spot coordinates 210 are stored in the calibration file in the device and are known a priori. These spot coordinates 210 are used as the basis for the dot mask creation step S21. The dot mask 211 is created from these spot coordinates 210.

[0156] The dot mask 211 consists of integer values at the spot coordinates and NaNs elsewhere. This configuration allows the dot mask 211 to clearly indicate the locations of the spots, which are the regions expected to contain the wanted (direct) signal plus any disturbance (global) signal. The dot mask 211 is used for visualization and spot location.

[0157] Fig. 4B shows an illustration of a dot mask 211 overlay ed onto a sensor array 50. Arrows indicate an x-axis and a y-axis as indicated. Pixels are arranged for example in a chequerboard pattern along the x-axis and the y-axis. A number of spot coordinates 30 are projected onto the sensor array 50. Only a few spot coordinates 30 are indicated for legibility. Sony Semiconductor Solutions Corporation et al.

[0158] The spot coordinates 30 are stored in a calibration file in the ToF camera and are known a priori. The dot mask 211 is created from the spot coordinates 30. The dot mask 211 consists of integer values at the spot coordinates 30 and NaNs (i.e. undefined values) elsewhere. The dot mask 211 indicates the locations of the spots, which are the regions expected to contain the wanted (direct) signal plus any disturbance (global) signal.

[0159] Dot histogram and valley histogram post-processing

[0160] The following embodiments describe the post-processing steps used to implement the improved DGS algorithm.

[0161] Fig. 5 shows a detailed illustration of the dot histogram selection step S22 as shown in Fig. 3, which selects dot histograms for further processing and discards histograms that are not selected.

[0162] The dot histogram selection step S22 is based on the dot mask 211, which is, for example, a two- dimensional array based on which the spots are located, and histograms 220, which is, for example, a two-dimensional array from which the histogram selection for a dot is obtained. The input parameter is dot binning window size 301a, which defines the size of the square window that selects the dots 51 (dot pixels).

[0163] The output of the dot histogram selection step S22 is Hist dot list 211a, which is a list of all selected dot histograms 5 Ih obtained in the dot pixels 51 within the selection window 53 defined by the input parameters. Hist dot list 21 la is structured for example as a three-dimensional array.

[0164] A dot 51 is a region on the sensor array where the light emitted by the light source is directly received by the sensor. Each dot 51 typically corresponds to one or more pixels on the sensor array, and these pixels are expected to contain the wanted (direct) signal. The direct signal is the primary measurement of interest, representing the time-of-flight (ToF) data that indicates the distance between the sensor and the object being measured. Each pixel of a dot 51 produces a dot histogram based on the detected light, which shows the distribution of time delays between the emitted and received light pulses.

[0165] Fig. 6A shows a detail of a sensor array 50 with an example dot 51 distribution. Nine spots 30 are shown. Each spot 30 corresponds to one or more dot pixels 51. Each dot pixel producing one dot histogram. The center spot has a dot binning window 53, which here encompasses, for example, 3x3 dot pixels 51 centered on a spot coordinate 30. The 3x3 dot pixels 51 subtended by the dot binning window 53 produce 9 dot histograms. Sony Semiconductor Solutions Corporation et al.

[0166] The sensor array 50 is structured with an x-axis and y-axis to define the coordinates of the pixels. Each spot 39 represents a region on the sensor array 50 where light is detected, and each dot pixel 51 produces a histogram based on the detected light. The dot binning window 53 around the center dot 51, which is located at the spot coordinate 30, is used to group multiple pixels, in this case, a 3x3 pixel area. This selection process, by applying the dot binning window 53, corresponds to the dot histogram selection step S22.

[0167] Fig. 6B shows a detail of the sensor array 50 with another example spot 30 distribution. This configuration is an alternative to the subject matter of Fig. 6A. Here, one dot pixel 51 subtends the selection window 53. The dot pixel 51 it produces one dot histogram.

[0168] Use of the configuration of Fig. 6A as opposed to 6B is selected based on the input parameter dot binning window size 301a.

[0169] Fig. 7A shows nine dot histograms 5 Ih overlaid on each other, as may be output by the 3x3 dot pixels 51 subtended by the dot binning window 53 in Fig. 6A. The x-axis indicates the distance bin, and the y-axis indicates the amplitude of the histogram in terms of counts.

[0170] The nine dot histograms correspond to those inside the selection window and are output in a four-dimensional array hist dot list.

[0171] The x-axis represents the distance bins, which are intervals of measured distances. Each bin corresponds to a specific range of distances that the emitted light pulse has traveled to and from the object. The bins typically indicate bins of length, such as millimeters or centimeters, depending on the resolution of the time-of-fhght sensor. Alternatively, the bind may indicate the time elapsed since the measured light pulse was emitted. The distance can then be calculated based on the speed of light.

[0172] The y-axis represents the amplitude of the histogram, indicating the number of counts or occurrences for each distance bin. Each count represents the number of times a light pulse has been detected at a specific distance bin.

[0173] Fig. 7B shows an example of a single dot histogram, as it may be output by the 1x1 pixels subtended by the dot binning window 53 in Fig. 6B. The x-axis indicates the distance bin, and the y-axis indicates the amplitude of the histogram in terms of counts.

[0174] The dot histogram 5 Ih corresponds to the dot pixel 51 in the selection window 53 and is output in hist dot list

[0175] Fig. 8 shows the valley histogram selection step S28. The valley histogram selection step S28 takes as inputs dot mask 211, histograms 220, Valley _circle_steps 301b, and Sony Semiconductor Solutions Corporation et al.

[0176] Valley circle radius 301c. S28 outputs selected valley histograms 221a (histogram valley list 221a). The histogram valley list comprises one or more valley histograms 54h.

[0177] The valley histogram selection step S28 uses the dot mask 211, which is a

[0060]

[0080] array from which the spots location of the spots, i.e. the spot coordinates 30, are obtained, and histograms 220, which are structured as a three-dimensional array from which the selected histograms for valleys 54 are obtained. Histograms 220 comprise individual histograms for each pixel on the Time-of-Flight pixel array from which dot histograms and valley histograms are seleccted. The selecting is further based on input parameters, which include Valley circle steps 301b and and Valley circle radius 301c. Valley circle steps 301b defines the total number of pixels uniformly distributed along a circle centered at the spot coordinates. Valley circle radius 301c defines the radius of a circle centered at the spot coordinates from which the valley pixels are taken in terms of pixels. Together, these parameters define the number and geometric pattern of the of valleys 54 (valley pixels).

[0178] The output of the valley histogram selection step S28 is hist valley list 221a, which is a list of all valley histograms obtained from the valley pixels in the circle defined by the input parameters. Hist valley list 221a is structured for example as a three-dimensional array.

[0179] Fig. 9A shows a detail of sensor array 50 with an example distribution of four valley pixels 54. Nine spots 30 are indicated. The center spot 30 has a dot binning window 53, which here encompasses, for example, 1x1 dot pixels 51 centered. This means that, in the present figure, one dot histogram 5 Ih is produced for the center spot 30. Also indicated are four valley pixels 54, arranged in a circle around the center dot 51. The radius of the circle of valley pixels 54 and the number of valley pixels 54 are determined by the input parameters Valley circle steps 301b and Valley circle radius 301c as described in Fig. 8.

[0180] The sensor array 50 is structured with an x-axis and y-axis to define the coordinates of the pixels. Each dot 51 represents a region on the sensor array 50 where light is detected, and each pixel within these dots 51 produces a histogram based on the detected light. The dot binning window 53 around the center spot 30 is used to select a specific pixel area, in this case, a 1x1 pixel area, to focus the signal processing on the histogram produced by this pixel.

[0181] The radius of this circle and the number of valley pixels 54 are determined by the input parameters Valley _circle_steps 301b and Valley circle radius 301c. The valley pixels 54 are used to obtain histograms that represent the global component (disturbance or noise) in the signal, such as background noise, multipath interference, or other disturbances. These histograms from the valley pixels 54 can then be subtracted from the histograms of the dot pixels 51 to Sony Semiconductor Solutions Corporation et al. isolate the direct component (wanted signal). This configuration helps in improving the accuracy and reliability of the depth and signal-to-noise ratio (SNR) calculations by focusing on both the direct and global components of the signal.

[0182] Input parameters are selected a priori based on the spot size and spot separation, which depends on the characteristics of the transmitter (Tx) and receiver (Rx). The circle is centered at the spot coordinate 30. In Fig. 1, as described hereinabove, the VCSEL array 5 serves as the transmitter (Tx), emitting light pulses LI towards the scene, and the ToF receiver array 7 acts as the receiver (Rx), capturing the reflected light L2 and converting it into electrical signals. These signals are then processed to determine the time delay

[0183] Fig. 9B shows a detail of sensor array 50 with an example distribution with six valley pixels 54. The radius of the circle of valley pixels 54 and the number of valley pixels 54 are determined by the input parameters Valley circle steps 301b and Valley circle radius 301c as described in Fig. 8.

[0184] Fig. 9C shows a detail of sensor array 50 with another example distribution of six valley pixels 54, but with a larger radius from the dot pixel 51 at the center dot 51. The radius of the circle of valley pixels 54 and the number of valley pixels 54 are determined by the input parameters Valley circle steps 301b and Valley circle radius 301c as described in Fig. 8.

[0185] Fig. 9D shows a detail of sensor array 50 with an example distribution of eight valley pixels 54. The radius of the circle of valley pixels 54 and the number of valley pixels 54 are likewise determined by the input parameters Valley circle steps 301b and Valley circle radius 301c as described in Fig. 8.

[0186] Fig. 10 shows the histogram filtering step S23, in which a predetermined number of dot histograms and discards the rest. The histogram filtering step S23 takes as inputs selected dot histograms 211a and dot number to keep 301d. S23 outputs filtered dot histograms 211b (histogram dot list 211b).

[0187] The histogram filtering step S23 performs two tasks: First, it orders the histograms in the input list as a function of the two-norm distance between the dot histograms 5 Ih from the dot pixels 51 inside the selection window 53 and the dot histogram 5 Ih at the spot coordinate 30.

[0188] The two-norm distance between one dot histogram 5 Ih as shown in Fig. 10 and the dot histogram 5 Ih at the spot coordinate 30 with the histogram 5 Ih at the spot coordinate 30, expressed as a vector x on a bin-by-bin basis, and the dot histogram 5 Ih for which the distance Sony Semiconductor Solutions Corporation et al. should be calculated expressed as a vector y on a bin-by-bin basis, is calculated, for example according to the Formula:

[0189] ||x- ||2= ((xi-yi)2+(x2-y2)2+.. +(xn-yn)2)1 / 2

[0190] Where indices on the right side indicate the component (corresponding to a bin number) in each of the vectors x and y representing the histogram 5 Ih at the spot coordinate 30 and the dot histogram 5 Ih for which the distance should be calculated respectively. The index n is equal to the total number of bin per histogram 5 Ih.

[0191] The two-norm distance between the vectors v and w is zero if cos((p) is unity.

[0192] Second, from the ordered list of histograms, the histogram filtering step S23 selects a number of dot histograms 5 Ih corresponding to the number defined by the dot number to keep parameter 301d. The parameter dot number to keep is less than or equal to the square of the dot binning window size. The histogram at the dot coordinate is always the first histogram in the list. If dot number to keep is equal to 1, then the dot histogram 5 Ih to choose is the dot histogram 5 Ih at the spot coordinate.

[0193] The histogram filtering step S23 keeps (i.e. selects) dot number to keep histograms based on their distance to the histogram at the spot coordinate. The distance is defined as the 2-norm (i.e. the Euclidean distance) between the center histogram and the rest of the histograms in the window.

[0194] Fig. 11A shows an illustration of a plurality of dot histograms 5 Ih resulting from dots 51 subtending a selection window 53 as shown, for example, in Fig. 6A. This plurality of dot histograms 5 Ih is provided to histogram filtering step S23 as explained in Fig. 10. The method keeps dot number to keep 301d histograms based on their distance to the histogram at the spot coordinate.

[0195] Fig. 1 IB shows an illustration of the filtering of histogram filtering step S23 applied to the histograms shown in Fig. 11 A. Three histograms remain. The filtering process in step S23 selects dot number to keep 301d histograms based on their distance to the histogram at the spot coordinate.

[0196] It should be noted that the histograms may be selected based on other parameters, such as a signal-to noise ratio of the histograms or height of the peak instead of the distance of the histogram to the histogram at the spot coordinate. Sony Semiconductor Solutions Corporation et al.

[0197] Fig. 12B shows a detailed explanation of spatial average calculation step S24. Spatial average calculation step S24 takes dot histograms 211b (histogram dot list 211b) as input and outputs averaged dot histograms 211c (histogram dot list 211c). Fig. 12B shows a detailed explanation of spatial average calculation step S29. Spatial average calculation step S29 takes valley histograms 221b (histogram valley list 221b) as input and outputs averaged valley histograms 221c (histogram valley list 221c). The spatial averaging of histograms outputs histograms typically with less noise but also lower count. To calculate a spatial average, the method involves averaging the values of corresponding bins across multiple histograms within a defined spatial region. The spatial average is computed by summing the values of each bin across the selected histograms and then dividing by the number of histograms.

[0198] Fig. 13 A shows an illustrative input of either spatial average calculation step S24 or spatial average calculation step S29 as discussed in Fig. 12A and Fig. 12B. Shown are nine histograms which may either be dot histograms 211b (histogram dot list 211b) or valley histograms 221b (histogram valley list 221b).

[0199] Fig. 13B shows, as an output, one averaged histogram that is an average of the nine histograms of Fig. 13 A. This averaged histogram may be the output of either spatial average calculation step

[0200] 524 or spatial average calculation step S29, resulting in either dot histograms 211c (histogram dot list 211c) or valley histograms 221c (histogram valley list 221c).

[0201] Fig. 14A shows a detailed explanation of noise floor removal step S25. Noise floor removal step

[0202] 525 takes dot histograms 211c (histogram dot list 211c) as input and outputs dot histograms 21 Id (histogram dot list 21 Id), based on floor processing parameters 301e. The noise floor removal step S25 needs to be applied to dot histograms and valley histograms similarly. It takes as input the ‘dgs_noise_floor_processing’ parameter 301e (floor processing parameter 301e), which controls the method used. If set to ‘None’, the noise floor removal step S25 is bypassed. If set to ‘median’, it calculates the median in each 16-bin part of the histograms and subtracts the value from the histogram. If set to ‘mean’, it calculates the mean in each 16-bin part of the histograms and subtracts the value from the histogram. Negative counts are clipped to zero.

[0203] For example, with dgs_noise_floor_processing set to ‘median’, the process is as follows: hist dot = hist dot - median(hist dot), hist_dot= hist dot - median(hist dot).

[0204] With dgs_noise_floor_processing set to ‘mean’, the process is: hist dot = hist dot - mean(hist dot), hist dot = hist dot - mean(hist dot). Sony Semiconductor Solutions Corporation et al.

[0205] Wherein negative values are clipped (i.e. set) to zero.

[0206] Fig. 14B shows a detailed explanation of noise floor removal step S210. Noise floor removal step S210 takes valley histograms 221c (histogram valley list 221c) as input and outputs valley histograms 221d (histogram valley list 221d), based on floor processing parameter 301e.

[0207] Fig. 15A shows an illustration (not to scale) of histograms to which the mean noise floor removal according to noise floor removal step S25 and noise floor removal step S210 is applied. The mean line is merely an illustration. To calculate a mean, sum the values of all data points in a given range and then divide by the number of data points. For example, for a histogram segment hist[0: 16], the mean is calculated, for example, as mean(hist[0: 16]) = (sum(hist[0]) + sum(hist[l]) + ... + sum(hist

[0015] )) / 16.

[0208] Fig. 15B shows an illustration of histograms after the mean noise floor removal according to noise floor removal step S25 and noise floor removal step S210 is applied. This mean value is then subtracted from each corresponding bin in the histogram segment to remove the noise floor. Negative counts resulting from this subtraction are clipped to zero, resulting, for example, in the histogram shown in Fig. 15B

[0209] Fig. 16A shows an illustration of histograms to which the median noise floor removal according to noise floor removal step S25 and noise floor removal step S210 is applied. The median line is merely an illustration, assuming a global mean, but a running or local mean may be used instead. To calculate a median, first sort the values of all data points in a given range in ascending order. If the number of data points is odd, the median is the middle value. If the number of data points is even, the median is the average of the two middle values. For example, for a histogram segment hist[0 : 16], sort the values in hist[0 : 16] and then find the middle value or the average of the two middle values. This median value is then subtracted from each corresponding bin in the histogram segment to remove the noise floor. Negative counts resulting from this subtraction are clipped to zero.

[0210] Fig. 16B shows an illustration of histograms after the median noise floor removal according to noise floor removal step S25 and noise floor removal step S210 is applied.

[0211] Fig. 17 shows the scaling step S211, which scales the valley histograms with respect to the dot histograms before calculation of the residual histogram. The scaling step S211 takes as inputs valley histograms 221d (histogram valley list 221d) and scaling parameter 30 If and outputs scaled valley histograms 221e (histogram valley list 221e). The scaling parameter 30 If, also Sony Semiconductor Solutions Corporation et al. referred to as Valley scaling, is a number, for example 2, which will result in each histogram bin to be multiplied by a factor of 2, provided in 30 If.

[0212] The rationale behind the scaling step S211 is to address situations where the global signal is underestimated due to factors such as a low number of valley pixels or the nature of the disturbance. By applying a scaling factor, the subtraction of the disturbance becomes more aggressive and effective. The scaled valley histograms 22 le (histogram valley list 22 le) are calculated using the formula:

[0213] Hist_valley_scaled = Valley _scaling x hist_valley

[0214] Fig. 18 illustrates the scaling step S211 and shows one input histogram being scaled by a factor of 2 according to the scaling step S211, resulting in a scaled histogram.

[0215] This figure shows the input and output of the valley histogram scaling step S211 from an arbitrary selection of valley pixels 54 around a spot with two-dimensional coordinates on the ToF receiver array . The input valley histograms 221d (histogram valley list 221d) are scaled by the scaling parameter 30 If, which in this example is 2, resulting in the scaled valley histograms 22 le (histogram valley list 22 le).

[0216] Fig. 19A illustrates step offset subtraction step S26, which takes input dot histograms 21 Id (histogram dot list 21 Id) and offset parameter 301g, which specifies an offset for dot histograms, and outputs dot histograms 212. The offset is specified a priori and subtracted from the dot histograms. However, a dynamic allocation may be provided. Depending on the intended use, the offset may be added instead.

[0217] Adding an offset to the histogram makes the later subtraction more effective by reducing bias due to large noise floor values. Large noise floor values are typical in strong sunlight scenes, where the noise's temporal characteristics make the DGS subtraction less effective. Negative values are clipped to zero.

[0218] Fig. 19B illustrates step the offset addition step S212, which takes input valley histograms 221e (histogram valley list 221e) and offset parameter 301h, which specifies an offset for valley histograms, and outputs valley histograms 222. The offset is specified a priori and added to the valley histograms. However, a dynamic allocation may be provided. Depending on the intended use, the offset may be subtracted instead.

[0219] Adding an offset to the histogram makes the later subtraction more effective by reducing bias due to large noise floor values. Large noise floor values are typical in strong sunlight scenes, where the noise's temporal characteristics make the DGS subtraction less effective. Sony Semiconductor Solutions Corporation et al.

[0220] Here, a simple offset is added or subtracted, however, a value related to the noise floor may instead be used. Negative values are clipped to zero.

[0221] Fig. 20 illustrates offsetting a histogram as required by the Offset addition step S212 and the offset subtraction step S26. Shown is an input histogram and an offset histogram. The offset histogram is calculated by subtracting the offset from each bin of the input histogram. Negative values are clipped to zero. The figure shows a subtraction of an offset as an example, but the offset may also be added instead of subtracted.

[0222] Fig. 21 describes step the histogram subtraction step S213, which uses the preprocessed dot histograms and valley histograms to calculate improved direct-global-separated histograms.

[0223] The inputs to the histogram subtraction step S213 are dot histogram 5 Ih (comprised in dot histograms 212) and valley histogram 54h (valley histograms 222). The output of this histogram subtraction step S213 is the subtraction of the valley histogram 54h from the dot histogram 5 Ih (i.e. the residual histogram): hist dgs = hist dot- hist valley scaled

[0224] In the hist dgs (DGS histograms 230), negative counts are clipped to zero, i.e., hist_dgs[hist_dgs < 0] = 0.

[0225] Fig. 22A illustrates the input histograms for S213. The input histograms are the dot histograms 212 as may be output by S26 and the valley histograms 222 as may be output by S212. The histograms shown in Fig. 22A correspond to a frame with strong interference. The target (direct) signal peak is the one generating counts in, for example, bin 12 (corresponding to a plane at ~lm). Fig. 21 A shows for example, that the dot histogram 5 Ih contains the target (direct) and disturbance (global) signals, with a target peak in, for example, bin 12 and an interference peak at, for example, bin 24. The valley histogram 54h contains mainly the disturbance (global) signal, with an interference peak at, for example, bin 24. Here, as an example, no noise floor processing (refer to Fig. 14, Fig. 15 and Fig. 16) was provided. The valley histograms were scaled by a factor 2.

[0226] Fig. 22B illustrates the output for S213, which is DGS histograms 230 (i.e., a residual histogram) for the input shown in Fig. 22A.

[0227] The target (direct) signal peak is the one generating counts in, for example, bin 12 (corresponding to a plane at ~lm). The intention of the left figure is to show that the dot histogram 5 Ih contains the target (direct) and disturbance (global) signals, with a target peak in, for example, bin 12 and an interference peak at, for example, bin 24. The valley histogram 54h Sony Semiconductor Solutions Corporation et al. contains mainly the disturbance (global) signal, with an interference peak at, for example, bin 24. The residual histogram 230 shown in Fig. 22B contains only the wanted (direct) signal; by subtracting the dot and valley histograms, a histogram with only the target peak is achieved.

[0228] Fig. 23 A illustrates the input histograms for the histogram subtraction step S213, which calculates a residual histogram 230 by subtracting the valley histograms 222 from the sot histograms 212.

[0229] The target (direct) signal peak is the one generating counts in, for example, bin 12 (corresponding to a plane at ~lm). The intention of the left figure is to show that the dot histogram 5 Ih contains the target (direct) and disturbance (global) signals, with a target peak in, for example, bin 12 and a weak interference peak at, for example, bin 24. The valley histogram 54h contains mainly the disturbance (global) signal, with a weak interference peak at, for example, bin 24.

[0230] Fig. 23B illustrates the output for S213, which is residual histogram 230 (i.e., DGS histogram 230) for the input shown in Fig. 23 A.

[0231] Fig. 24A shows an input histogram for S213, which includes dot histograms 212 as output by S26 and valley histograms 222 as output by S212. The noise floor indicated in Fig. 24A corresponds to sunlight at 9.6Klux.

[0232] Fig. 24B illustrates the output for S213, which is a residual histogram 230 (i.e., DGS histogram 230) for the input shown in Fig. 24A. The parameters used are dgs_noise_floor_processing set to None and valley _scaling set to 1.

[0233] Fig. 25A shows an input histogram for S213, which includes dot histograms 212 as output by S26 and valley histograms 222 as output by S212. The noise floor indicated in Fig. 25A corresponds to sunlight at 9.6Klux. The parameters used are dgs_noise_floor_processing set to median and valley scaling set to 1.

[0234] Fig. 25B illustrates the output for S213, which is a residual histogram 230 (i.e., a DGS histogram 230) for the input shown in Fig. 25A.

[0235] Depth and SNR calculation

[0236] Fig. 26 shows step SI 14, which uses DGS histograms 230 and an SNR threshold 30 If to calculate DGS Depth 105a and DGS SNR 105b according to dToF.

[0237] The input to this block is a histogram, which can be from hist dgs, hist dot, or other sources. The output of this block includes depth and SNR values, which are calculated specifically at the Sony Semiconductor Solutions Corporation et al. spot coordinates. The depth values are masked by the SNR threshold 30 If to ensure that only reliable measurements are considered.

[0238] In this step, the residual histograms 230, resulting from the histogram subtraction step S213, are processed using standard dToF depth calculation equations, such as peak detection and CoG.

[0239] The depth calculation involves determining the time delay between the emission and reception of light pulses, which is directly proportional to the distance of the object from the sensor. This is achieved by analyzing the peaks in the DGS histograms 230. The depth information is then computed based on these time-of-flight values.

[0240] The SNR calculation measures the quality of the signal by comparing the amplitude of the direct component to the amplitude of the global component. A higher SNR indicates a clearer measurement, while a lower SNR suggests the presence of noise. The calculated depth and SNR values are structured as arrays, representing the depth and SNR at the spot coordinates.

[0241] Example output of an improved DGS algorithm

[0242] Fig. 27A shows ToF receiver array 50 with a plurality of dots 51, of which only two are indicated for legibility. Also shown are the x-axis, referring to pixels arranged along the x-axis of ToF receiver array 50, and the corresponding y-axis. Greyscale according to the color bar CB indicates depth dgs. Fig. 27A is obtained for an SNR threshold of 6. This illustrates an output of the Depth and SNR calculation step SI 12.

[0243] Depth dgs values are shown in the spot coordinates. This figure shows the depth output of the software datapath 1 Iwith different snr thresholds. In the case of snr threshold = 6, there are more depth values that are masked out than at lower values of the snr thresholds, as will be shown in Fig. 27B.

[0244] Fig. 27B shows ToF receiver array 50 with a plurality of dots 51, of which only two are indicated for legibility. Also shown are the x-axis, referring to pixels arranged along the x-axis of ToF receiver array 50, and the corresponding y-axis. Greyscale according to the color bar CB indicates depth dgs. Fig. 27B is obtained for an SNR threshold of 0. This illustrates an output of Depth and SNR calculation step SI 12.

[0245] Depth dgs values are shown in the spot coordinates. This figure shows the depth output of the software datapath 1 Iwith different snr thresholds. In the case of snr threshold = 0, there are more depth values visible compared to Fig. 27A.

[0246] Dynamic of dot and valley selection Sony Semiconductor Solutions Corporation et al.

[0247] Described below is an alternative embodiment that uses, instead of the selection Step S22 or the selection step S28, a dynamic selection of dots 51 and valleys 54 based on a noise metric is performed.

[0248] Fig. 28A shows an alternative, dynamic, selection of dots. The method comprises a noise metric determination step S22A wherein, as a noise metric, an SNR 302 is obtained, and a dynamic selection step S22B wherein, based on histograms 220, dot mask 211, SNR 302, and input parameters 301, dots are selected according to the SNR 302. Step S22B outputs selected dot histograms 5 Ih as histogram dot list 211a.

[0249] In this alternative method, step S22A determines the noise metric, such as SNR 302, for the dots and valleys. Based on this noise metric, step S22B dynamically selects the dots and valleys. This dynamic selection is based on the histograms 220, dot mask 211, SNR 302, and input parameters 301. The dynamic selection step S22B aims to optimize the configuration of dots and valleys to achieve the best possible signal quality.

[0250] Fig. 28B shows an alternative, dynamic, selection of valleys. The method comprises a noise metric determination step S22A wherein, as a noise metric, an SNR 302 is obtained, and a dynamic selection step S22B wherein, based on histograms 220, dot mask 211, SNR 302, and input parameters 301, valleys 54 are selected according to the SNR 302. Step S22B outputs selected valley histograms 221a (histogram valley list 221a).

[0251] Selecting the optimal dot and valley configuration (i.e., number and arrangement of dots and valley pixels) is necessary because the optimal configuration changes depending on the distance to an object and / or sunlight level. By checking the SNR (or other noise metric), the method can dynamically decide the optimal configuration.

[0252] Fig. 29 illustrates the selecting of dynamic selection steps S22A and S22B. The figure shows three heat maps 50a, 50b, and 50c with selection according to decreasing SNR. The decreasing SNR is indicated by an arrow.

[0253] Heat map 50a has nine dots 51, heat map 50b has five dots 51, and heat map 50c has one dot 51, the dots 51 being dot pixels subtended by the selection window 53. The heat maps indicate the intensity of incident light on a ToF sensor array 50.

[0254] Selecting the optimal dot and valley configuration (i.e., number and arrangement of dots and valley pixels) is necessary because the optimal configuration changes depending on the distance to an object and / or sunlight level. By checking the SNR (or other noise metric), the method can dynamically decide the optimal configuration. Sony Semiconductor Solutions Corporation et al.

[0255] The method comprises a noise metric determination step S22A shown in Fig. 28A and Fig. 28B wherein, as a noise metric, an SNR 302 is obtained. Based on this noise metric, a dynamic selection step S22B is performed wherein, based on histograms 220, dot mask 211, SNR 302, and input parameters 301, dots are selected according to the SNR 302.

[0256] Implementation example

[0257] In Fig. 29, heat map 50a shows a configuration with nine dots 51 and a valley radius of 3, suitable for high SNR conditions. As the SNR decreases, heat map 50b shows a configuration with five dots 51 and a valley radius of 2. In the lowest SNR condition, heat map 50c shows a configuration with one dot 51 and a valley radius of 2.

[0258] Fig. 30 shows a circuitry 1200 comprising the following components: CPU 1201, image sensor 1210, cellular communications interface 1211, user interface 1212, ToF interface 1213, storage 1202, RAM 1203, Bluetooth 1204, and WLAN 1205.

[0259] The CPU 1201 is responsible for executing instructions and processing data for the overall operation of the device. It manages and coordinates the activities of all other components within the circuitry 1200. The CPU 1201 processes the data obtained from the ToF sensor 1 (Fig. 1) and other inputs, running algorithms such as the Direct and Global Separation (DGS) algorithm described in the application.

[0260] The image sensor 1210 captures visual information from the environment, converting light into electronic signals that can be processed to create images or videos. This component is used for applications that require visual data, such as photography or video recording. In the context of the application, the image sensor 1210 can work in conjunction with the ToF sensor 1 (Fig. 1) to provide enhanced depth information and 3D imaging capabilities.

[0261] The cellular communications interface 1211 enables the device to connect to cellular networks, allowing for voice calls, text messaging, and data transmission over mobile networks. This component is essential for maintaining connectivity when Wi-Fi is not available, ensuring that the device can transmit ToF data and other information to remote servers or other devices.

[0262] The user interface 1212 provides the means for users to interact with the device. This can include touchscreens, buttons, and other input / output mechanisms that allow users to control the device and receive feedback. The user interface 1212 is used for configuring the ToF sensor settings, viewing depth maps, and interacting with applications that utilize ToF data.

[0263] The ToF interface 1213 is specifically designed to handle data from the Time-of-Flight (ToF) sensor 1 (Fig. 1). It processes the ToF data to determine depth information and create 3D Sony Semiconductor Solutions Corporation et al. representations of the environment. This interface is used for applications that require accurate distance measurements, such as augmented reality, robotics, and autonomous vehicles. The ToF interface 1213 manages the data acquisition through the dToF system 10 as described hereinabove.

[0264] The storage 1202 component is used to store data and applications. It retains information even when the device is powered off, providing long-term storage for the operating system, applications, and user data. In the context of the application, storage 1202 is used to save ToF raw data 200 (Fig. 2), processed histograms, depth maps, and configuration files such as depth, csv.

[0265] The RAM 1203 (Random Access Memory) is used for temporary data storage while the device is in operation. It provides fast access to data that the CPU 1201 needs to execute tasks, improving the overall performance and responsiveness of the device. RAM 1203 is essential for handling the large amounts of data generated by the ToF sensor 1 (Fig. 1) and for running the complex algorithms described in the application, such as the DGS algorithm Si l l (Fig. 2).

[0266] The Bluetooth 1204 component enables wireless communication with other Bluetooth-enabled devices. This allows for the exchange of data over short distances, facilitating connections with peripherals such as headphones, keyboards, and other mobile devices. Bluetooth 1204 can be used to transmit ToF data to nearby devices for further processing or display.

[0267] The WLAN 1205 (Wireless Local Area Network) component allows the device to connect to Wi-Fi networks. This provides high-speed internet access and enables data transmission within a local network, supporting activities such as web browsing, streaming, and online communication. WLAN 1205 is used for transmitting ToF data to cloud servers or other devices for real-time processing and analysis.

[0268] All units and entities described in this specification and claimed in the appended claims can, if not stated otherwise, be implemented as integrated circuit logic, for example on a chip, and functionality provided by such units and entities can, if not stated otherwise, be implemented by software.

[0269] In so far as the embodiments of the disclosure described above are implemented, at least in part, using software-controlled data processing apparatus, it will be appreciated that a computer program providing such software control and a transmission, storage or other medium by which such a computer program is provided are envisaged as aspects of the present disclosure.

[0270] Note that the present technology can also be configured as described below. Sony Semiconductor Solutions Corporation et al.

[0271] (1) An electronic device, comprising circuitry 1200 configured to determine Time-of-Flight histogram data 220 from raw data obtained by a Time-of-Flight pixel array 50; generate, based on spot coordinates of spots received by the Time-of-Flight pixel array 50, a dot mask 211, the dot mask being indicative of dot pixels 51; select based on the dot mask 211, one or more dot pixels 51; select based on the dot mask 211 and based on a predetermined geometrical pattern, one or more valley pixels 53, and calculate one or more residual histograms 230 by subtracting one or more valley histograms 54h, generated by the valley pixels 54, from one or more dot histograms 5 Ih, generated by the dot pixels 51; and generate depth information 105 based on the one or more residual histograms 230.

[0272] (2) An electronic device, comprising circuitry 1200 configured to determine Time-of-Flight histogram data 220 from raw data obtained by a Time-of-Flight pixel array 50; generate, based on spot coordinates of spots received by the Time-of-Flight pixel array 50, a dot mask 211, the dot mask being indicative of dot pixels 51; select based on the dot mask 211, one or more dot pixels 51; select based on the dot mask, one or more valley pixels 53, and subtract a noise floor from the one or more valley histograms 54h and / or from the one or more dot histograms (5 Ih), and calculate one or more residual histograms 230 by subtracting one or more valley histograms 54h, generated by the valley pixels 54, from one or more dot histograms 5 Ih, generated by the dot pixels 51; and generate depth information 105 based on the one or more residual histograms 230.

[0273] (3) An electronic device, comprising circuitry 1200 configured to determine Time-of-Flight histogram data 220 from raw data obtained by a Time-of-Flight pixel array 50; generate, based on spot coordinates of spots received by the Time-of-Flight pixel array 50, a dot mask 211, the dot mask being indicative of dot pixels 51; select based on the dot mask 211, one or more dot pixels 51; select based on the dot mask, one or more valley pixels 53, and perform offset processing on the one or more valley histograms 54h and / or from the one or more dot histograms 5 Ih, and calculate one or more residual histograms 230 by subtracting one or more valley histograms 54h, generated by the valley pixels 54, from one or more dot histograms 5 Ih, generated by the dot pixels 51; and generate depth information 105 based on the one or more residual histograms 230.

[0274] (4) An electronic device, comprising circuitry 1200 configured to determine Time-of-Flight histogram data 220 from raw data obtained by a Time-of-Flight pixel array 50; generate, based on spot coordinates of spots received by the Time-of-Flight pixel array 50, a dot mask 211, the dot mask being indicative of dot pixels 51; select based on the dot mask 211, one or more dot Sony Semiconductor Solutions Corporation et al. pixels 51; select based on the dot mask, one or more valley pixels 53, and filter the one or more dot histograms 5 Ih based on a two-norm distance from a dot histogram 5 Ih located at a spot coordinate 30, and / or and filter the one or more dot histograms 5 Ih based on a peak value, and calculate one or more residual histograms 230 by subtracting one or more valley histograms 54h, generated by the valley pixels 54, from one or more dot histograms 5 Ih, generated by the dot pixels 51; and generate depth information 105 based on the one or more residual histograms 230.

[0275] (5) The electronic device 1 according to any one of (1) to (4), wherein the circuitry 1200 is further configured to select the one or more valley pixels 54 based on the predetermined set of coordinates on Time-of Flight pixel array such that the selected valley pixels 54 are arranged in a predetermined geometrical pattern on the Time-of-Flight pixel array 50.

[0276] (6) The electronic device 1 according to (5), wherein the predetermined geometrical pattern is selectable from among a plurality of selectable geometrical patterns.

[0277] (7) The electronic device 1 according to (6), wherein the plurality of selectable geometrical patterns are selectable based on pattern parameters.

[0278] (8) The electronic device 1 according to (6), wherein the circuitry 1200 is further configured to select the predetermined geometrical pattern based on a noise metric 302.

[0279] (9) The electronic device 1 according to (5), wherein the predetermined geometrical pattern is a circle centered on one of the spot coordinates 30.

[0280] (10) The electronic device 1 according to (9), wherein the predetermined geometrical pattern is a circle with a predetermined radius.

[0281] (11) The electronic device 1 according to (10), wherein the one or more valley pixels 54 are a predetermined number of valley pixels 54 arranged regularly along the circle.

[0282] (12) The electronic device 1 according to (10), wherein the circuitry 1200 is further configured to select the predetermined number of valley pixels 54 based on a noise metric 302.

[0283] (13) The electronic device 1 according to any one of (1) to (4), wherein the circuitry 1200 is further configured to determine a noise floor of the one or more valley histograms 54h; and subtract the noise floor of the one or more v911ey histograms 54h from the one or more valley histograms 54h prior to calculating the one or more residual histograms 230.

[0284] (14) The electronic device 1 according to (13), wherein the noise floor is a running mean of the valley histogram 54h; or wherein the noise floor is a running median of the valley histogram 54h. Sony Semiconductor Solutions Corporation et al.

[0285] (15) The electronic device 1 according to any one of (1) to (4), wherein the circuitry 1200 is further configured to spatially average the one or more valley histograms 54h prior to calculating the one or more residual histograms 230.

[0286] (16) The electronic device 1 according to any one of (1) to (4), wherein the circuitry 1200 is further configured to add a predetermined valley histogram offset to the one or more valley histograms 54h prior to calculating the one or more residual histograms 230.

[0287] (17) The electronic device 1 according to any one of (1) to (4), wherein the circuitry 1200 is further configured to scale the one or more valley histograms 54h by a predetermined scaling factor prior to calculating the one or more residual histograms 230.

[0288] (18) The electronic device 1 according to any one of (1) to (4), wherein the circuitry 1200 is further configured to select the one or more dot pixels 51 based on whether a dot pixel 51 is inside or outside of a selection window 53.

[0289] (19) The electronic device 1 according to any one of (1) to (4), wherein the selection window 53 is a circle with a predetermined radius centered on one of the spot coordinates 30.

[0290] (20) The electronic device 1 according to (19), wherein the predetermined radius is selectable based on selection window parameters.

[0291] (21) The electronic device 1 according to (19), wherein the circuitry 1200 is further configured to select the predetermined radius based on a noise metric 302.

[0292] (22) The electronic device 1 according to any one of (1) to (4), wherein the circuitry 1200 is further configured to select, prior to calculating the one or more residual histograms 230, from the one or more dot histograms 5 Ih, a predetermined number of dot histograms 5 Ih having a lower two-norm distance with respect to a histogram corresponding to a spot coordinate 30 of the one of the spots received by the Time-of-Flight receiver array 50 than all other dot histograms

[0293] 5 Ih of the one or more dot histograms 5 Ih.

[0294] (23) The electronic device 1 according to any one of (1) to (4), wherein the circuitry 1200 is further configured to select, prior to calculating the one or more residual histograms 230, from the one or more dot histograms 5 Ih, a predetermined number of dot histograms 5 Ih having a higher peak value than all other dot histograms 5 Ih of the one or more dot histograms 5 Ih.

[0295] (24) The electronic device 1 according to any one of (1) to (4), wherein the circuitry 1200 is further configured to determine a noise floor of the dot histogram 5 Ih; and subtract the noise floor of the dot histogram 5 Ih from the dot histogram 5 Ih prior to calculating the one or more residual histograms 230. Sony Semiconductor Solutions Corporation et al.

[0296] (25) The electronic device 1 according to (24), wherein the noise floor is a running mean of the dot histogram 5 Ih; or wherein the noise floor is a running median of the dot histogram 51h.

[0297] (26) The electronic device 1 according to any one of (1) to (4), wherein the circuitry 1200 is further configured to spatially average the one or more dot histograms 5 Ih prior to calculating the one or more residual histograms 230.

[0298] (27) The electronic device 1 according to any one of (1) to (4), wherein the circuitry 1200 is further configured to subtract a predetermined dot histogram offset from the one or more dot histograms 5 Ih prior to calculating the one or more residual histograms 230.

[0299] (28) A sensing method for a time-of-flight sensing device, comprising determining Time-of- Flight histogram data from raw data obtained by a Time-of-Flight pixel array 50; generating based on spot coordinates of spots received by the Time-of-Flight pixel array 50, a dot mask, the dot mask 211 being indicative of dot pixels 51; selecting based on the dot mask 211, one or more dot pixels 51; selecting based on the dot mask 211 and based on a predetermined geometrical pattern, one or more valley pixels 53, and calculating one or more residual histograms 230 by subtracting one or more valley histograms 54h, generated by the valley pixels 54, from one or more dot histograms 5 Ih, generated by the dot pixels 51; and generating depth information 105 based on the one or more residual histograms 230.

[0300] (29) A computer program that, if executed by a computer, causes the computer to apply the method according to (28).

[0301] (30) A non-transitory computer-readable recording medium that stores therein a computer program product, which, when executed by a processor, causes the method according to (29) to be performed.

[0302] (31) An electronic device, comprising circuitry 1200 configured to determine Time-of-Flight histogram data 220 from raw data obtained by a Time-of-Flight pixel array 50; generate, based searching, in the histogram data 220, for one or more local maxima of an intensity of the light received by the Time-of-Flight pixel array 50, a dot mask 211, the dot mask being indicative of dot pixels 51; select based on the dot mask 211, one or more dot pixels 51; select based on the dot mask 211 and based on a predetermined geometrical pattern, one or more valley pixels 53, and calculate one or more residual histograms 230 by subtracting one or more valley histograms 54h, generated by the valley pixels 54, from one or more dot histograms 5 Ih, generated by the dot pixels 51; and generate depth information 105 based on the one or more residual histograms 230. Sony Semiconductor Solutions Corporation et al.

[0303] LIST OF REFERENCE SIGNS

[0304] LI Emitted light

[0305] L2 Reflected light

[0306] 1 ToF sensor

[0307] 5 VCSEL array

[0308] 7 ToF receiver array

[0309] 8 Control circuitry

[0310] 10 dToF system

[0311] 11 Software datapath

[0312] 30 Spot coordinate

[0313] 50 ToF receiver array

[0314] 51 Dot pixel

[0315] 51h Dot histogram

[0316] 53 Selection window

[0317] 54 Valley pixel

[0318] 54h Valley histogram

[0319] 105 DGS Depth / DGS SNR (primary output)

[0320] 105a DGS Depth

[0321] 105b DGS SNR

[0322] 110 Flood output (secondary output)

[0323] 200 raw data

[0324] 210 Spot coordinates

[0325] 211 Dot mask

[0326] 211a-e Histogram dot list

[0327] 212 Dot histograms

[0328] 220 Histograms Sony Semiconductor Solutions Corporation et al.

[0329] 221a-f Histogram valley list

[0330] 222 Valley histograms

[0331] 230 DGS histograms

[0332] 301 Input parameters

[0333] 301a-i Input parameters

[0334] 302 SNR

[0335] 1200 Circuitry

[0336] 1210 Image sensor

[0337] 1211 Cellular communications interface

[0338] 1212 User interface

[0339] 1213 ToF interface

[0340] 1202 Storage

[0341] 1203 RAM

[0342] 1204 Bluetooth

[0343] 1205 WLAN

[0344] Si l l DGS step

[0345] SI 12 Depth and SNR calculation step

[0346] 521 Dot mask creation step

[0347] 522 Dot histogram selection step

[0348] 523 Histogram filtering step

[0349] 524 Spatial average calculation step

[0350] 525 Noise floor removal step

[0351] 526 Offset subtraction step

[0352] 528 Valley histogram selection step

[0353] 529 Spatial average calculation step

[0354] 5210 Noise floor removal step

[0355] 5211 Scaling step Sony Semiconductor Solutions Corporation et al.

[0356] 5212 Offset subtraction step

[0357] 5213 Histogram subtraction step

Claims

Sony Semiconductor Solutions Corporation et al.CLAIMS1. An electronic device (1), comprising circuitry (1200) configured to determine Time-of-Flight histogram data (220) from raw data obtained by a Time-of- Flight pixel array (50); generate (S21), based on spot coordinates of spots received by the Time-of-Flight pixel array (50), a dot mask (211), the dot mask being indicative of dot pixels (51); select (S22), based on the dot mask (211), one or more dot pixels (51); select (S28), based on the dot mask (211) and based on a predetermined set of coordinates on the Time-of Flight pixel array, one or more valley pixels (53), and calculate (S213) one or more residual histograms (230) by subtracting one or more valley histograms (54h), generated by the valley pixels (54), from one or more dot histograms (5 Ih), generated by the dot pixels (51); and generate (SI 12) depth information (105) based on the one or more residual histograms (230).

2. The electronic device (1) according to claim 1, wherein the circuitry (1200) is further configured to select (S28) the one or more valley pixels (54) based on the predetermined set of coordinates on Time-of Flight pixel array such that the selected valley pixels (54) are arranged in a predetermined geometrical pattern on the Time-of-Flight pixel array (50).

3. The electronic device (1) according to claim 2, wherein the predetermined geometrical pattern is selectable from among a plurality of selectable geometrical patterns.

4. The electronic device (1) according to claim 3, wherein the plurality of selectable geometrical patterns are selectable based on pattern parameters.

5. The electronic device (1) according to claim 3, wherein the circuitry (1200) is further configured to select (S28B) the predetermined geometrical pattern based on a noise metric (302).

6. The electronic device (1) according to claim 2, wherein the predetermined geometrical pattern is a circle centered on one of the spot coordinates (30).

7. The electronic device (1) according to claim 6, wherein the predetermined geometrical pattern is a circle with a predetermined radius.Sony Semiconductor Solutions Corporation et al.

8. The electronic device (1) according to claim 7, wherein the one or more valley pixels (54) are a predetermined number of valley pixels (54) arranged regularly along the circle.

9. The electronic device (1) according to claim 7, wherein the circuitry (1200) is further configured to select (S28B) the predetermined number of valley pixels (54) based on a noise metric (302).10 The electronic device (1) according to claim 1, wherein the circuitry (1200) is further configured to determine a noise floor of the one or more valley histograms (54h); and subtract (S210) the noise floor of the one or more valley histograms (54h) from the one or more valley histograms (54h) prior to calculating (S213) the one or more residual histograms (230).

11. The electronic device (1) according to claim 10, wherein the noise floor is a running mean of the valley histogram (54h); or wherein the noise floor is a running median of the valley histogram (54h).

12. The electronic device (1) according to claim 1, wherein the circuitry (1200) is further configured to spatially average (S29) the one or more valley histograms (54h) prior to calculating (S213) the one or more residual histograms (230).

13. The electronic device (1) according to claim 1, wherein the circuitry (1200) is further configured to add (S212) a predetermined valley histogram offset to the one or more valley histograms (54h) prior to calculating (S213) the one or more residual histograms (230).

14. The electronic device (1) according to claim 1, wherein the circuitry (1200) is further configured to scale (S211) the one or more valley histograms (54h) by a predetermined scaling factor prior to calculating (S213) the one or more residual histograms (230).

15. The electronic device (1) according to claim 1, wherein the circuitry (1200) is further configured to select (S22) the one or more dot pixels (51) based on whether a dot pixel (51) is inside or outside of a selection window (53).

16. The electronic device (1) according to claim 1, wherein the selection window (53) is a circle with a predetermined radius centered on one of the spot coordinates (30).

17. The electronic device (1) according to claim 16, wherein the predetermined radius is selectable based on selection window parameters.

18. The electronic device (1) according to claim 16, wherein the circuitry (1200) is further configured to select the predetermined radius based on a noise metric (302).Sony Semiconductor Solutions Corporation et al.

19. The electronic device (1) according to claim 1, wherein the circuitry (1200) is further configured to select (S23), prior to calculating (S213) the one or more residual histograms (230), from the one or more dot histograms (5 Ih), a predetermined number of dot histograms (5 Ih) having a lower two-norm distance with respect to a histogram corresponding to a spot coordinate (30) of the one of the spots received by the Time-of-Flight receiver array (50) than all other dot histograms (5 Ih) of the one or more dot histograms (5 Ih).

20. The electronic device (1) according to claim 1, wherein the circuitry (1200) is further configured to select (S23), prior to calculating (S213) the one or more residual histograms (230), from the one or more dot histograms (5 Ih), a predetermined number of dot histograms (5 Ih) having a higher peak value than all other dot histograms (5 Ih) of the one or more dot histograms (51h).

21. The electronic device (1) according to claim 1, wherein the circuitry (1200) is further configured to determine a noise floor of the dot histogram (5 Ih); and subtract the noise floor of the dot histogram (5 Ih) from the dot histogram (5 Ih) prior to calculating the one or more residual histograms (230).

22. The electronic device (1) according to claim 21, wherein the noise floor is a running mean of the dot histogram (5 Ih); or wherein the noise floor is a running median of the dot histogram (5 Ih).

23. The electronic device (1) according to claims 1, wherein the circuitry (1200) is further configured to spatially average the one or more dot histograms (5 Ih) prior to calculating (S213) the one or more residual histograms (230).

24. The electronic device (1) according to claim 1, wherein the circuitry (1200) is further configured to subtract (S26) a predetermined dot histogram offset from the one or more dot histograms (5 Ih) prior to calculating (S213) the one or more residual histograms (230).

25. A sensing method for a time-of-flight sensing device, comprising determining Time-of-Flight histogram data from raw data obtained by a Time-of-Flight pixel array; generating (S21), based on spot coordinates of spots received by the Time-of-Flight pixel array, a dot mask, the dot mask (211) being indicative of dot pixels (51); selecting (S22), based on the dot mask (211), one or more dot pixels (51);Sony Semiconductor Solutions Corporation et al. selecting (S28), based on the dot mask (211) and based on a predetermined set of coordinates on Time-of Flight pixel array, one or more valley pixels (53), and calculating (S213) one or more residual histograms (230) by subtracting one or more valley histograms (54h), generated by the valley pixels (54), from one or more dot histograms (5 Ih), generated by the dot pixels (51); and generating (SI 12) depth information (105) based on the one or more residual histograms (230).

26. A computer program that, if executed by a computer, causes the computer to apply the method according to claim 25.

Images