Object detection device
The object detection device addresses false detections by extracting shadow candidate regions and calculating size changes based on light source movement, improving accuracy in distinguishing low three-dimensional objects from road surfaces.
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- ASTEMO LTD
- Filing Date
- 2026-01-15
- Publication Date
- 2026-07-23
AI Technical Summary
Existing object detection systems face challenges in accurately distinguishing low three-dimensional objects from road surfaces due to small differences in measurement and interference from unrelated light sources or objects, leading to false detections.
An object detection device that extracts shadow candidate regions, calculates the size change of these regions based on light source movement, and determines the presence of three-dimensional objects by comparing actual and estimated shadow changes, using multiple light sources including vehicle headlights and infrastructure lights.
The device effectively suppresses false detections by accurately identifying three-dimensional objects despite interference from unrelated light sources, enhancing detection accuracy.
Smart Images
Figure JP2026001022_23072026_PF_FP_ABST
Abstract
Description
Object detection device
[0001] The present invention relates to an object detection device.
[0002] In an object detection system using an in-vehicle camera, an object detection device that detects relatively low three-dimensional objects such as fallen objects and tires on the road is known. As a means for detecting any low three-dimensional object, object detection by grouping three-dimensional point clouds using three-dimensional point cloud measurement by a stereo camera can be mentioned. However, the difference between the low three-dimensional object and the three-dimensional points measured on the road surface is small, making it difficult to separate from measurement errors, and there has been a problem of misidentifying a flat surface such as road paint as a low three-dimensional object.
[0003] As a detection technique for low three-dimensional objects using information other than three-dimensional measurement information, there is JP 2014-72681 (Patent Document 1). In this publication, it is described as a problem that "since an object is detected based on a shadow generated by light irradiation from one direction, there may be a case where almost no shadow is generated depending on the direction in which the object is irradiated with light." And as a solution, it is described that "in a first aspect of the present invention, an acquisition unit that acquires a captured image from a capturing unit that captures an object to generate a captured image, a control unit that controls an irradiation unit that irradiates the object with a plurality of lights from different directions, a recognition unit that recognizes an image of a shadow generated by the irradiation unit irradiating the object with a plurality of lights in the captured image, and a detection unit that detects an object based at least on the shadow recognized by the recognition unit are provided."
[0004] JP 2014-72681 A
[0005] In object detection using a light source and a shadow, a light source was installed at a known position, and an image was captured while switching the installed light source, and object detection was performed by regarding the area where a change occurred as a shadow.
[0006] However, when performing object detection in various scenes such as an in-vehicle camera, there is a possibility that light sources existing outside this system such as the lighting of other vehicles or shadows caused by objects other than the detection target may be reflected. That is, there has been a problem that false detection due to disturbance may occur.
[0007] The objective of the present invention is to provide an object detection device that suppresses false detections caused by the influence of unrelated light sources or objects.
[0008] To achieve the above objective, the present invention includes: a shadow candidate region extraction unit that extracts shadow candidates, which are candidate regions in which the shadow of a three-dimensional object illuminated by a light source is captured, from an image captured by a camera installed on a vehicle; a region size calculation unit that determines the amount of change in the size of the shadow candidate region due to the movement or change in position of the light source; and a three-dimensional object detection unit that determines whether or not a three-dimensional object exists that forms the shadow candidate region based on the amount of change in the size of the shadow candidate region.
[0009] According to the present invention, it is possible to provide an object detection device that can suppress false detections caused by the influence of unrelated light sources or objects.
[0010] Further features related to the present invention will become apparent from the description herein and the accompanying drawings. Problems, configurations, and effects not described above will be revealed by the following description of embodiments.
[0011] Hardware configuration diagram of the object detection device. Functional block diagram explaining the internal functions of the image processing unit. Functional block diagram explaining the internal functions of the region size calculation unit. Diagram explaining a specific example of calculation processing by the region size calculation unit. Diagram showing an example of changes in the candidate shadow area on the road surface. Functional block diagram explaining the internal functions of the region size calculation unit in Example 2. Diagram explaining the case where the headlights of an oncoming vehicle are used as a light source. Diagram showing an example of changes in the candidate shadow area on the road surface. Functional block diagram explaining the internal functions of the 3D object detection unit in Example 2. Diagram to explain an example of selection conditions. Diagram explaining an example of a combination of light source and candidate shadow area by the combination determination unit.
[0012] Embodiments of the present invention will be described below with reference to the drawings.
[0013] [Example 1] Figure 1 is a hardware configuration diagram of an object detection device. The object detection device 101 in this embodiment is mounted on the vehicle and detects objects present on the road surface in front of the vehicle, and in particular detects relatively low-height three-dimensional objects such as tires. The vehicle is equipped with a camera system 102 that captures images of the area in front of it. The camera system 102 has, for example, one or more on-board cameras 103 that can capture images of the area in front of the vehicle. The cameras 103 are composed of, for example, a stereo camera or a monocular camera.
[0014] The object detection device 101 is connected to the camera system 102 and acquires images captured by the camera system 102. Note that the object detection device 101 is not limited to a configuration separate from the camera system 102 as in this embodiment; for example, it may be integrated into the camera system 102.
[0015] The object detection device 101 is composed of, for example, an electronic control unit (ECU) and includes components such as a memory 104 (RAM or ROM), a CPU 105, and an external output unit 107. These components are connected to each other via a communication line 108. The CPU 105 executes the calculation processing of the image processing unit 201, which will be described below, according to the instructions of the program stored in the memory 104. The object detection device 101 realizes the following functions of the image processing unit 201 by executing the program stored in the memory 104 using the CPU 105.
[0016] The image processing unit 201 uses the image captured by the camera 103 to detect three-dimensional objects through various processing steps. The specific object detection process will be described later. The object detection results are transmitted externally from the object detection device 101 via the external output unit 107 and used by other functions of the vehicle, such as preventative safety control. These functions may be incorporated as part of the image processing unit.
[0017] The components of the image processing unit will be described below. Figure 2 is a functional block diagram illustrating the internal functions of the image processing unit. As shown in Figure 2, the image processing unit 201 has a shadow candidate region extraction unit 210, a region size calculation unit 211, and a three-dimensional object detection unit 212 as internal functions.
[0018] The shadow candidate region extraction unit 210 performs a process to extract areas in the image where it is estimated that the shadow of a three-dimensional object is captured, as shadow candidate regions. For example, the shadow candidate region extraction unit 210 extracts areas in the image with a lower brightness value compared to the surrounding area as shadow candidate regions.
[0019] In this embodiment, since it is desirable that the shadows of three-dimensional objects be projected onto the road surface, a process may be performed to set the road surface area as a processing area from the image using a method such as Semantic Segmentation before brightness evaluation, and to extract candidate shadow areas from within that set processing area. However, it is difficult to distinguish between the shadows of three-dimensional objects and stains or black paint on the road surface from image texture alone. At this point, it is also acceptable to include false detection results other than shadows in the calculation of candidates.
[0020] Figure 3 is a functional block diagram illustrating the internal functions of the domain size calculation unit, and Figure 4 is a diagram illustrating the content of the calculation process performed by the domain size calculation unit.
[0021] The region size calculation unit 211 estimates what changes will occur in the actual shadow when the light source is set to two different points, assuming that there is a shadow generated when a three-dimensional object is illuminated by a light source. As shown in Figure 3, the region size calculation unit 211 includes a light source movement estimation unit 301 and a shadow region change estimation unit 302 as internal functions.
[0022] The light source movement estimation unit 301 estimates the behavior of the vehicle 401 using, for example, CAN information such as vehicle speed and yaw rate. When the headlights 403 of the vehicle 401 are used as the light source, the amount of light source movement can be estimated by using vehicle operation information such as the amount of vehicle movement and vehicle speed of the vehicle 401 over a certain period of time.
[0023] The shadow area change estimation unit 302 estimates the change in the candidate shadow area based on the obtained light source movement amount. The change in the candidate shadow area is the change in the size of the candidate shadow area extracted by the candidate shadow area extraction unit 210, and can be determined based on the change in length and direction of the candidate shadow area. For example, as shown in Figures 4(1) and (2), consider the case where the vehicle 401 travels in a straight line from point A to point B on the road surface 411 for a distance of 421, and the headlight 403 generates shadows of the three-dimensional object 412 at points A and B, respectively. In this case, as shown in Figure 4(2), the lengths s and t of the candidate shadow areas 413 and 414 are given by the following relation (1) using the distance Z to the three-dimensional object 412 and the movement amount d of the vehicle 401.
[0024] In addition, the direction in which the shadow extends on the image (di, dj) can be expressed on the image as follows, using equations (2) and (3), where (X, Z) is the horizontal position and depth of the three-dimensional object 412, and (X_light, Z_light) is the horizontal position and depth of the headlight. In the above formula, H is the camera installation height, f is the camera focal length, and wi and wj are the camera pixel pitches.
[0025] Here, the amount of light source movement can also be determined by estimating the distance from the vehicle to the candidate shadow area based on the image. For example, one method is to estimate the distance to the object by assuming that the lower end of the detected object is in contact with the road surface, or to measure the distance using the principle of triangulation with a stereo camera system using two or more cameras. Using this information, the amount of light source movement can be estimated, and from the estimated amount of light source movement, the change in the size of the candidate shadow area can be determined.
[0026] Furthermore, even without the vehicle itself moving, it is possible to capture an image of an object illuminated by two light sources with known movement amounts by sequentially illuminating light sources located at different positions. Possible means of utilizing light sources at different positions include the left and right headlights of the vehicle 401, or external light sources such as streetlights in conjunction with infrastructure equipment, as controllable light sources with known grounding positions. In other words, when the vehicle or infrastructure equipment has multiple light sources whose positional relationship is known and which are sequentially illuminated in synchronization with the camera's imaging, the area size calculation unit 211 can determine the amount of light source movement based on the position of each light source illuminated during imaging and calculate the amount of change in the size of the candidate shadow area. For example, these methods can be used when the vehicle has a configuration that allows the left and right headlights to be illuminated sequentially, or when there are multiple infrastructure lights such as streetlights connected to a network.
[0027] Returning to the explanation of Figure 2, the 3D object detection unit 212 compares the actual change in region size calculated from the length t of the candidate shadow region 414 extracted by the candidate shadow region extraction unit 210 with the estimated values of the direction θ and length t of the shadow, which are calculated by the region size calculation unit 211 as the change in the candidate shadow region due to the movement of the light source (change in the position of the light source) using equations (2) and (3).
[0028] Then, if the change in the actual size of the candidate shadow area matches the estimated values of the shadow direction θ and length t, the object is determined to be an object capable of casting a shadow and is detected as a three-dimensional object. In reality, noise has an effect, so if the difference between the change in the actual size of the candidate shadow area and the estimated values of the shadow direction θ and length t is below a threshold, it is considered to match.
[0029] Figure 5 shows an example of the change in the shadow candidate area on the road surface. When the vehicle 401 is at point A, the shadow candidate area 413 extends away from the headlight 403, with the object 412 in between, due to the light 501 from the vehicle's headlight 403. As the vehicle 401 moves straight from point A along the Z direction and approaches the object 412, the length of the shadow candidate area 413 shortens, and at point B it becomes the shadow candidate area 414. The shadow area change amount estimation unit 302 estimates the amount of change from the shadow candidate area 413 to the shadow candidate area 414 using equations (1) to (3).
[0030] According to the object detection device 101 of the embodiment described above, the device estimates how the shadow area will change based on the change in the position of the light source, compares this with the actual change in the candidate shadow area, and determines that it is a three-dimensional object if they match. This suppresses false detections caused by the influence of unrelated light sources or objects.
[0031] [Example 2] Next, Example 2 of the present invention will be described. In Example 2, the same reference numerals are used for components similar to those in Example 1, and their detailed descriptions will be omitted.
[0032] Example 2 is a modification of Example 1 and shows an embodiment in which a light source other than the light source mounted on the vehicle (hereinafter referred to as "other light sources") is used. According to this embodiment, it is possible to utilize more light sources, and it is also possible to use light sources located in a position where the shadow candidate area extends from the three-dimensional object toward the vehicle for processing. As a result, changes in the area can be observed with greater accuracy, and detection accuracy can be improved.
[0033] Figure 6 is a functional block diagram illustrating the internal functions of the region size calculation unit in this embodiment. In addition to the light source movement estimation unit 301 and shadow region change estimation unit 302 of Embodiment 1, the region size calculation unit 211 of this embodiment includes a light source detection unit 601, a position measurement unit 602, and a headlight control unit 603.
[0034] The light source detection unit 601 detects other light sources from the image acquired by the camera system 102. As a detection method, for example, one method is to detect a small area with a particularly high brightness value within the image as a single light source. In ordinary object detection images, overexposure occurs around the light source due to saturation of brightness values, which can lead to detection failure. Therefore, it is desirable to use an image that does not experience overexposure even in bright areas through HDR processing or the like. Unlike the light source of the vehicle's headlights in Embodiment 1, it is not possible to estimate the movement of other light sources from the vehicle's behavior, so the position measurement unit 602 is used to measure the position of the light source.
[0035] The position measurement unit 602 measures the position of other light sources detected by the light source detection unit 601. If the other light source is, for example, the headlight of another vehicle, the position measurement unit 602 estimates the distance by means of estimating the distance to the object from the image coordinates of the point where the lower end of the other vehicle is installed on the road. If the other light source is a light source such as a streetlamp, where it is difficult to estimate the ground contact position, the distance can be estimated by the same process by assuming the height from the road surface. Alternatively, the distance to the other light source is measured based on multiple images captured by multiple cameras placed spaced apart on the vehicle. For example, a stereo camera system using two or more cameras is more suitable for this embodiment because it can achieve high-precision three-dimensional measurement of the other light source. Using these distance measurement results, the light source movement amount estimation unit 301 calculates the amount of movement of the other light source.
[0036] The light source movement estimation unit 301, when multiple other light sources are present, determines the amount of change in the size of the shadow candidate region for each of the multiple other light sources based on the positions of these multiple other light sources measured by the position measurement unit 602. The three-dimensional object detection unit 212 determines whether or not a three-dimensional object exists that forms the shadow candidate region, based on the amount of change in the size of the shadow candidate region determined for at least one of the other light sources.
[0037] Figure 7 shows an example of using the headlights of an oncoming vehicle as a light source. Figure 7(1) is a schematic diagram of image 721 taken from the vehicle itself at a predetermined time, and Figure 7(2) is a schematic diagram of image 722 taken after a predetermined time has elapsed from the time shown in Figure 7(1).
[0038] Images 721 and 722 show the approaching oncoming vehicle 701. On the road surface 711, candidate shadow regions 713 and 714 of the three-dimensional object 712 are formed by light from another light source, the headlights 703 of the oncoming vehicle 701. As shown in Figure 7, the candidate shadow regions 713 and 714 of the three-dimensional object 712 change as the oncoming vehicle 701 moves. The change in the length of the candidate shadow region at this time can be calculated in the same way as equation (1) in Example 1.
[0039] The headlight control unit 603 adjusts the brightness of the vehicle's headlights 403 in synchronization with the imaging timing of the camera 103 when there are other light sources measured by the position measurement unit 602. When the shadow candidate area extraction unit 210 detects shadows generated by other light sources such as the headlights 703 of an oncoming vehicle 701, if the vehicle's headlights 403 strongly illuminate the shadow candidate area, there is a risk of the shadow disappearing.
[0040] Therefore, when capturing images for this process, by performing controls such as turning off the headlights 403 of the vehicle 401, changing the illumination level, or avoiding illumination in areas where shadows are generated through light distribution control, the shadow candidate area extraction unit 210 can extract shadow candidate areas with greater accuracy.
[0041] Figure 8 shows an example of the change in the shadow candidate area on the road surface. At point C, the shadow candidate area 813 extends in the direction approaching the camera 103 with the three-dimensional object 812 in between, due to the light 804 from the headlights 803 (another light source) of another vehicle 801. As the other vehicle 801 approaches the three-dimensional object 812, the length of the shadow candidate area 813 shortens, and at point D it becomes the shadow candidate area 814. The shadow area change amount estimation unit 302 shown in Figure 6 estimates the amount of change from the shadow candidate area 813 to the shadow candidate area 814 using the above-mentioned equations (1) to (3).
[0042] Next, the case where there are a plurality of light sources and a plurality of shadow candidate regions will be described. The shadow candidate region extraction unit 210 estimates a plurality of shadow candidate regions 414, and based on their shapes and positional relationships, performs association (pairing) with the plurality of other detected light sources. Then, the region size calculation unit 211 obtains the amount of change in the size of each shadow candidate region accompanying the movement of the other light source associated with the shadow candidate region. The three-dimensional object detection unit 212 selects a shadow candidate region and a light source appropriate for the process of three-dimensional object detection and performs three-dimensional object detection.
[0043] FIG. 9 is a functional block diagram for explaining the internal functions of the three-dimensional object detection unit in the present embodiment. The three-dimensional object detection unit 212 includes a shadow selection unit 901, a light source selection unit 902, a combination determination unit 903, and a change amount comparison unit 904.
[0044] In the present embodiment, since there may be a mixture of a plurality of light sources other than the host vehicle, it is necessary to select appropriate ones from the shadow candidate regions and light sources for three-dimensional object detection. The shadow selection unit 901 selects a shadow candidate region to be used for subsequent processing from among the plurality of shadow candidate regions. Those with a large difference in luminance value from the surroundings or those with a strong edge portion of the shadow candidate region are appropriate because the amount of change in the shadow candidate region can be estimated accurately.
[0045] The light source selection unit 902 selects a light source to be used for subsequent processing from the plurality of detected light sources. FIG. 10 is a diagram for explaining an example of the selection conditions. In the example shown in FIG. 10, the host vehicle 1011 is traveling in the right lane 1002 of two lanes 1002 and 1003 traveling in the same direction on the road 1001. In the left lane 1003, a vehicle 1051 traveling in the forward direction is traveling, and in the right lane 1002 where the host vehicle 1011 is traveling, a three-dimensional object 1006 such as a dropped object with a relatively low height has fallen. And in the right lane 1004 of the oncoming lane, two oncoming vehicles 1021 and 1041 are traveling, and in the left lane 1005 of the oncoming lane, an oncoming vehicle 1031 is traveling.
[0046] The light source selection unit 902 selects the headlight of an appropriate other vehicle from the oncoming vehicles 1021, 1031, and 1041 with respect to the host vehicle 1011 and the three-dimensional object 1006. The position of the three-dimensional object 1006 may be input from the outside as an object detection result that may be misdetected by combining it with the vicinity of the shadow candidate region or an existing object detection function. Since it is necessary to extract the shadow candidate region and observe the change in its size, it is desirable that the shadow is generated closer to the host vehicle 1011.
[0047] That is, the oncoming vehicle 1021 existing closer to the host vehicle 1051 or the three-dimensional object 1006 is excluded. When comparing the oncoming vehicles 1031 and 1041, since light such as the headlight diffuses according to the distance, a light source closer to the three-dimensional object 1006 generates a darker shadow on the road surface. Therefore, the light source selection unit 902 selects the headlight of the oncoming vehicle 1041 closer to the three-dimensional object 1006 as an appropriate light source. In the above-described embodiment, an example of selecting one light source has been described, but a plurality of light sources may be selected.
[0048] The combination determination unit 903 determines which of the shadow candidate region and the light source selected by the shadow selection unit 901 and the light source selection unit 902 should be combined to compare the amount of change. As a criterion, the intensity of the light source, the darkness of the shadow, and the absolute size of the shadow candidate region assumed from the light source position information are used. That is, by using the luminance values of the light source and the shadow candidate region on the image to select a brighter light source and a darker shadow candidate region, and by using a light source with a larger shadow candidate region assumed from the light source position information for the processing, the accuracy of this processing using the shadow candidate region can be improved. The combination determination unit 903 uses only pairs in which the light source is located farther from the shadow candidate region as pairs of the light source and the shadow candidate region.
[0049] FIG. 11 is a diagram for explaining an example of combination of a light source and a shadow candidate region by the combination determination unit. Light 1102, 1112, and 1122 are respectively incident on a three-dimensional object 1120 from three light sources 1101, 1111, and 1121, and shadow candidate regions 1103, 1113, and 1123 are respectively formed. The three light sources 1101, 1111, and 1121 and the shadow candidate regions 1103, 1113, and 1123 are combined as pairs, respectively.
[0050] Finally, the change amount comparison unit 904 compares the change amounts of the light source and the shadow candidate region combined by the combination determination unit 903, and performs 3D object detection processing in the same manner as in Example 1.
[0051] According to the embodiment described above, a more stable object detection device 101 can be provided that uses multiple light sources other than the light source mounted on the vehicle 1011.
[0052] It should be noted that the present invention is not limited to the embodiments described above, and various modifications are included. For example, the embodiments described above are described in detail to make the present invention easier to understand, and are not necessarily limited to those having all the configurations described. Furthermore, it is possible to replace parts of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add configurations from other embodiments to the configuration of one embodiment. In addition, it is possible to add, delete, or replace parts of the configuration of each embodiment with other configurations.
[0053] Furthermore, each of the above configurations may be implemented either partially or entirely in hardware, or through program execution on a processor. Also, the control lines and information lines shown are those deemed necessary for illustrative purposes and do not necessarily represent all control lines and information lines in the actual product. In practice, almost all configurations can be considered interconnected.
[0054] 101...Object detection device 102...Camera system 201...Image processing unit 210...Shadow candidate area extraction unit 211...Area size calculation unit 212...3D object detection unit 301...Light source movement amount estimation unit 302...Shadow area change amount estimation unit 601...Light source detection unit 602...Position measurement unit 603...Headlight control unit 901...Shadow selection unit 902...Light source selection unit 903...Combination determination unit 904...Change amount comparison unit
Claims
1. An object detection device comprising: a shadow candidate region extraction unit that extracts a shadow candidate region, which is a candidate region in which the shadow of a three-dimensional object illuminated by a light source is captured from an image captured by a camera installed on a vehicle; a region size calculation unit that determines the amount of change in the size of the shadow candidate region due to the movement or change in position of the light source; and a three-dimensional object detection unit that determines whether or not a three-dimensional object exists that forms the shadow candidate region based on the amount of change in the size of the shadow candidate region.
2. An object detection device according to claim 1, wherein, when the headlights of the vehicle are used as the light source, the amount of movement of the light source is estimated using the vehicle's operation information, and the amount of change in the size of the candidate shadow area is determined from the amount of movement of the light source.
3. An object detection device according to claim 1, characterized in that it estimates the distance from the vehicle to the candidate shadow area based on the image, and when the headlights of the vehicle are used as the light source, it estimates the amount of movement of the light source using the estimated change in distance, and determines the size of the candidate shadow area from the amount of movement of the light source.
4. An object detection device according to claim 1, wherein the region size calculation unit includes a position measurement unit for measuring the position of the light source, and when there are multiple light sources, the amount of change in the size of the shadow candidate region is determined for each of the multiple light sources based on the position of the light source measured by the position measurement unit, and the three-dimensional object detection unit determines whether or not there is a three-dimensional object that forms the shadow candidate region based on the amount of change in the size of the shadow candidate region determined for at least one light source.
5. An object detection device according to claim 4, wherein the position measuring unit measures the distance to the light source based on a plurality of images captured by a plurality of cameras arranged spaced apart on the vehicle.
6. An object detection device according to claim 4, wherein the shadow candidate region extraction unit estimates a plurality of shadow candidate regions and associates them with a plurality of detected light sources based on their shape and positional relationship, and the region size calculation unit determines the amount of change in the size of the shadow candidate region due to the movement of the light source associated with the shadow candidate region.
7. An object detection device according to claim 4, characterized in that it comprises a headlight control unit that adjusts the brightness of the vehicle's headlights in synchronization with the imaging timing of the camera when there is a light source measured by the position measuring unit.
8. An object detection device according to claim 4, wherein the three-dimensional object detection unit selects a combination of a candidate shadow region and a light source to be used for determination using the shape of the candidate shadow region and the position of the light source.
9. An object detection device according to claim 1, comprising a plurality of light sources whose positional relationships are known and which are sequentially lit in synchronization with camera imaging, wherein the region size calculation unit determines the amount of light source movement based on the position of each light source lit during imaging, and determines the amount of change in the size of the candidate shadow region.
10. An object detection device according to claim 9, characterized in that it uses the left and right headlights of a vehicle as a plurality of light sources.
11. An object detection device according to claim 9, characterized in that it uses network-connected infrastructure lighting as a plurality of light sources.
12. An object detection device according to claim 4, characterized in that only pairs of light sources and shadow candidate regions are used in which the light source is located at a distance from the shadow candidate region.