Determination device, determination method, and recording medium

Abstract

This determination device for determining at least recognition reliability of an object in front of a vehicle: acquires a stereo image including the object in front of the vehicle; generates, on the basis of the stereo image, a parallax image indicating parallax to the object; sets, in the parallax image, a determination region including a point corresponding to a road boundary object in front of the vehicle; and determines the recognition reliability of the object on the basis of an effective parallax ratio of the number of effective pixel blocks in which the parallax is effective among the total number of pixel blocks to the total number of pixel blocks included in the determination region.

Description

Determination device, determination method, and recording medium

[0001] This disclosure relates to a determination device, a determination method, and a recording medium. In particular, this disclosure relates to a determination device, a determination method, and a recording medium for determining at least the reliability of recognizing an object in front of a vehicle.

[0002] Conventionally, there is a known technology that uses parallax obtained from stereo images to recognize objects in front of a vehicle.

[0003] For example, Patent Document 1 discloses a road edge detection method that includes obtaining a parallax diagram including a road area and a corresponding V-parallax diagram, extracting road lines Lv from the V-parallax diagram, obtaining a sub-U-parallax diagram including roads based on the road lines Lv extracted from the V-parallax diagram, and self-adaptively extracting road edges from the sub-U-parallax diagram.

[0004] Japanese Patent Publication No. 2014-175007

[0005] According to the conventional technology disclosed in Patent Document 1, in environments with poor visibility such as snow, rain, or fog, there was a risk that the position of road boundary objects such as curbs or side walls would be detected in a position different from the actual position. In other words, in environments with poor visibility, the stereo image captured by the stereo camera may become unclear. In such cases, if the position of road boundary objects such as curbs or side walls is calculated using the parallax obtained from the stereo image, there was a risk that the position of the road boundary objects would be detected in a position different from the actual position.

[0006] In light of these circumstances, the purpose of this disclosure is to provide a technology that suppresses misrecognition of road boundary objects by stereo cameras in environments with poor visibility.

[0007] A determination device according to one embodiment of the present disclosure is a determination device for determining the recognition reliability of an object in front of a vehicle, comprising one or more processors and one or more memories connected to the one or more processors, wherein the one or more processors acquire a stereo image including the object in front, generate a parallax image showing the parallax to the object based on the stereo image, set a determination region in the parallax image including a point corresponding to a road boundary object in front, and determine the recognition reliability based on the effective parallax ratio of the number of effective pixel blocks in which the parallax is effective out of the total number of pixel blocks to the total number of pixel blocks included in the determination region.

[0008] A determination method according to one embodiment of the present disclosure is a determination method for determining at least the reliability of recognition of an object in front of a vehicle, comprising: a computer acquiring a stereo image including the object in front of the vehicle; generating a parallax image showing the parallax to the object based on the stereo image; setting a determination region in the parallax image that includes points corresponding to road boundary objects in front of the vehicle; and determining the reliability of recognition based on the effective parallax ratio of the number of effective pixel blocks in which the parallax is effective out of the total number of pixel blocks to the total number of pixel blocks included in the determination region.

[0009] A recording medium according to one embodiment of the present disclosure is a non-temporary tangible recording medium on which a determination program for determining the reliability of recognition of an object in front of a vehicle is recorded, wherein the recording medium contains a determination program that causes a computer to acquire a stereo image including the object in front of the vehicle, generate a parallax image showing the parallax to the object based on the stereo image, set a determination region in the parallax image that includes a point corresponding to a road boundary object in front of the vehicle, and determine the reliability of recognition based on the effective parallax ratio of the number of effective pixel blocks in which the parallax is effective out of the total number of pixel blocks to the total number of pixel blocks included in the determination region.

[0010] According to one embodiment of this disclosure, misrecognition of road boundary objects by a stereo camera can be suppressed even in environments with poor visibility.

[0011] This is a schematic diagram showing an example of the configuration of a vehicle equipped with a determination device according to one embodiment of this disclosure. This is a block diagram showing an example of the configuration of a determination device according to one embodiment of this disclosure. This is a diagram illustrating a parallax image. This is a diagram illustrating a determination area set in a parallax image. This is a flowchart illustrating an example of the operation of a determination device according to one embodiment of this disclosure.

[0012] Preferred embodiments of this disclosure will be described in detail below with reference to the attached drawings. In this specification and the drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant descriptions will be omitted.

[0013] (1. Vehicle) Referring to Figure 1, Vehicle 1 is configured as a four-wheeled automobile that transmits the drive torque output from the drive source 2 to the wheels. Vehicle 1 may be an automobile equipped with an internal combustion engine such as a gasoline engine or a diesel engine as the drive source 2, or it may be an electric vehicle equipped with a drive motor as the drive source 2. Examples of electric vehicles include BEV (Battery Electric Vehicle), HEV (Hybrid Electric Vehicle), PHEV (Plug-in Hybrid Electric Vehicle), or FCEV (Fuel Cell Electric Vehicle).

[0014] The drive source 2 outputs drive torque which is transmitted to the front wheel drive shaft 4F via a transmission (not shown) and a differential mechanism 3. The drive of the drive source 2 and the transmission is controlled by a control device 7, which will be described later. The combination of drive wheels and the driving method are not particularly limited, and the vehicle 1 may be a front-wheel drive vehicle, a rear-wheel drive vehicle, or a four-wheel drive vehicle. Furthermore, if the vehicle 1 is configured as an electric vehicle, it may be an electric vehicle equipped with a drive motor corresponding to each wheel.

[0015] In addition to the above-described driving force source 2, the vehicle 1 is at least provided with a steering device 5, a braking device 6, and a control device 7 as devices used for driving control.

[0016] The steering device 5 is provided on the front wheel drive shaft 4F. The steering device 5 may include an electric motor (not shown) and a gear mechanism (not shown), and adjusts the steering angle of the front wheels by being controlled by the control device 7.

[0017] The braking device 6 applies a braking force to the wheels by being controlled by the control device 7. When the vehicle 1 is configured as an electric vehicle, the braking device 6 may be used in combination with the regenerative brake by the drive motor as the driving force source 2.

[0018] The control device 7 includes at least one or more ECUs (Electronic Control Units) that respectively control the driving of the driving force source 2, the steering device 5, and the braking device 6.

[0019] During manual driving, the control device 7 calculates the driving torque to be output to the driving force source 2 based on the operation amount of an accelerator pedal (not shown) by the driver. Also, during manual driving, the control device 7 calculates the braking force to be output to the braking device 6 based on the operation amount of a brake pedal (not shown) by the driver. Further, during manual driving, the control device 7 controls the steering device 5 based on the steering angle of the steering wheel 8 by the driver.

[0020] During automatic driving, the control device 7 controls the drive power source 2, the steering device 5, and the brake device 6 based on the target vehicle speed, the target steering angle, etc. of the vehicle 1 that are appropriately set using known or arbitrary driving support technologies. However, the control device 7 is connected to a determination device 20 described later via a dedicated line or communication means such as CAN (Controller Area Network) or LIN (Local Inter Net), and may perform control such as causing the vehicle 1 to retreat to the road shoulder according to the determination result of the determination device 20. Note that as the driving support technology, an advanced driver-assistance system (ADAS: Advanced Driver-Assistance Systems) is exemplified, but the present disclosure is not limited thereto.

[0021] The vehicle 1 further includes a stereo camera 10, a vehicle state sensor 11, and a position detection sensor 12.

[0022] The stereo camera 10 includes a main camera 10a and a sub camera 10b. The main camera 10a and the sub camera 10b each include an imaging element such as a CCD (Charged Coupled Devices) or a CMOS (Complementary Metal Oxide Semiconductor). The main camera 10a and the sub camera 10b are each connected to the determination device 20 by wired or wireless communication means, and are connected to the control device 7 as necessary. The main camera 10a and the sub camera 10b each capture the front of the vehicle 1 to generate a stereo image, and transmit the generated stereo image to the determination device 20 and, as necessary, to the control device 7. In FIG. 1, the main camera 10a and the sub camera 10b may be configured with the left and right reversed.

[0023] Specifically, the main camera 10a generates a reference image at a predetermined frame rate, and the sub-camera 10b generates a comparison image at a predetermined frame rate. As will be described in detail later, the determination device 20 acquires the reference image generated by the main camera 10a and the comparison image generated by the sub-camera 10b as a stereo image. Furthermore, the determination device 20 generates a parallax image showing the parallax to an object included in the stereo image by applying image processing such as stereo matching to the acquired stereo image. The distance to the object included in the stereo image can be determined from the parallax image.

[0024] In addition to the stereo camera 10, vehicle 1 may also be equipped with a camera (not shown) for imaging the area behind vehicle 1. Furthermore, vehicle 1 may also be equipped with one or more distance measuring sensors (not shown) from among radar sensors such as LiDAR (Light Detection And Ranging) or millimeter-wave radar and ultrasonic sensors.

[0025] The vehicle state sensor 11 is a known or arbitrary sensor for detecting the state of the vehicle 1. For example, the vehicle state sensor 11 may include a vehicle speed sensor for detecting the speed of the vehicle 1, and may include a steering angle sensor for detecting the steering angle of the steering wheel 8. The vehicle state sensor 11 transmits the detection result to the control device 7 and, if necessary, to the determination device 20. The detection result from the vehicle state sensor 11 is used as appropriate in the driving assistance of the vehicle 1 by the control device 7.

[0026] The position detection sensor 12 may include a GNSS (Global Navigation Satellite System) sensor that receives satellite signals from positioning satellites such as GPS (Global Positioning System) satellites. The position detection sensor 12 transmits information indicating the current position of the vehicle 1, which is included in the received satellite signals, to the control device 7, and to the determination device 20 as necessary.

[0027] In addition, vehicle 1 may further include a known or arbitrary HMI (Human Machine Interface) 13 that presents various information to the driver of vehicle 1 through image or text display or audio output, etc.

[0028] (2. Determination Device) The determination device 20 according to this embodiment will be described in detail with reference to Figure 2.

[0029] (2-1. Configuration Example) The determination device 20 functions as a device that determines at least the reliability of recognizing an object in front of the vehicle 1 by having one or more CPUs (Central Processing Units) or other processors execute a computer program. The computer program is a computer program that causes the processor to execute the operations that the determination device 20 should perform, which will be described later. The computer program executed by the processor may be recorded on a recording medium that functions as a memory unit 22, which will be described later, or it may be recorded on a recording medium built into the determination device 20 or on any recording medium that can be attached externally to the determination device 20.

[0030] The recording medium for storing computer programs may include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs, DVDs, and Blu-ray®; magneto-optical media such as floppy disks; memory elements such as RAM (Random Access Memory) and ROM (Read Only Memory); flash memory such as USB (Universal Serial Bus) memory and SSD (Solid State Drive); and other media capable of storing programs.

[0031] As described above, the determination device 20 is connected to at least the control device 7 and the stereo camera 10. It is also possible that some or all of the components of the determination device 20 are provided on the control device 7 side, or that they are provided on the server (not shown) side capable of communicating with the vehicle 1.

[0032] The determination device 20 comprises at least a processing unit 21 and a storage unit 22. If some or all of the components of the determination device 20 are provided on the server side, the determination device 20 further comprises a communication unit (not shown) which includes a communication interface for communicating with the vehicle 1 via a known or arbitrary network (not shown).

[0033] (Processing Unit) The processing unit 21 comprises one or more processors such as CPUs and various peripheral components. Part or all of the processing unit 21 may consist of updatable components such as firmware, or it may be a program module that is executed by instructions from the CPU, etc.

[0034] (Storage Unit) The storage unit 22 is composed of one or more storage elements such as RAM or ROM that are connected to the processing unit 21 in a communicative manner. However, the type and number of storage units 22 are not particularly limited. The storage unit 22 stores information such as computer programs executed by the processing unit 21, various parameters used in arithmetic processing, detection results, and calculation results.

[0035] (2-2. Functional Configuration of the Processing Unit) The functional configuration of the processing unit 21 of the determination device 20 will be described below. The processing unit 21 includes an image acquisition unit 211, an image generation unit 212, a region setting unit 213, a ratio calculation unit 214, and a determination unit 215. Each of the image acquisition unit 211, image generation unit 212, region setting unit 213, ratio calculation unit 214, and determination unit 215 is a function realized by the execution of a computer program by one or more processors such as a CPU. However, some or all of the image acquisition unit 211, image generation unit 212, region setting unit 213, ratio calculation unit 214, and determination unit 215 may be configured using analog circuits.

[0036] (Image Acquisition Unit) The image acquisition unit 211 acquires a stereo image including an object in front of the vehicle 1. Specifically, the image acquisition unit 211 may acquire a reference image including an object in front of the vehicle 1, captured by the main camera 10a included in the stereo camera 10, and a comparison image including an object in front of the vehicle 1, captured by the sub-camera 10b included in the stereo camera 10, as a stereo image. It is preferable that the image acquisition unit 211 acquires the stereo image at a calculation cycle synchronized with the frame rate of the stereo camera 10, and the acquired stereo image may be stored in the storage unit 22 linked to the acquisition time.

[0037] (Image Generation Unit) The image generation unit 212 generates a parallax image showing the parallax to an object in front of the vehicle 1 based on the stereo image acquired by the image acquisition unit 211. Specifically, the image generation unit 212 applies known or arbitrary stereo matching processing to the reference image and comparison image as stereo images. More specifically, the image generation unit 212 searches the comparison image for blocks corresponding to blocks arbitrarily extracted from the reference image, and calculates the parallax of the blocks in the reference image from the positional relationship between the blocks in the reference image and the blocks in the comparison image. It is preferable that the image generation unit 212 generates a parallax image, for example, shown in Figure 3, by performing this series of processing on all blocks included in the reference image. It is preferable that the image generation unit 212 generates a parallax image each time the image acquisition unit 211 acquires a stereo image, and the generated parallax image may be stored in the storage unit 22 in association with the acquisition time of the stereo image. Here, each block of the parallax image is colored according to the parallax, and for example, a color other than black may mean that the parallax is effective. Although the disparity image shown in Figure 3 is illustrated as black and white binary data for convenience, it is actually multi-level data that includes color information. However, this disclosure is not limited to this, and each block of the disparity image may be associated with a value corresponding to the disparity, and if that value is above a threshold, it may mean that the disparity is effective.

[0038] (Region Setting Unit) The region setting unit 213 sets a determination region in the disparity image generated by the image generation unit 212 that includes points corresponding to road boundary objects in front of the vehicle 1. Examples of road boundary objects include curbs and side walls, but this disclosure is not limited to these and may also include snow walls, etc.

[0039] The area setting unit 213 may set multiple determination areas as determination areas, each corresponding to a plurality of points separated from each other along the road boundary object in front of the vehicle 1, on at least one of the left and right sides in the direction of travel of the vehicle 1. This allows the determination unit 215, described later, to perform a determination even if some of the plurality of determination areas become unclear due to the lens surface of the stereo camera 10 or deposits on the windshield of the vehicle 1 on which the stereo camera 10 is installed. Therefore, misrecognition of road boundary objects by the stereo camera 10 can be suppressed even in environments with poor visibility.

[0040] Specifically, the region setting unit 213 may set the following determination regions in the disparity image generated by the image generation unit 212: a determination region including a point 9m (Z=9m) in front of the vehicle 1, a determination region including a point 13m (Z=13m) in front of the vehicle 1, and a determination region including a point 21m (Z=21m) in front of the vehicle 1, among the road boundary objects in front of the vehicle 1. Note that 9m in front is an example of a point where the area in front of the vehicle 1 can be imaged without being affected by the length of the front nose which depends on the type of vehicle 1 and the installation position of the stereo camera 10. However, this disclosure is not limited thereto, and the position, spacing, and number of determination regions can be appropriately set according to the processing time required by the driver assistance system.

[0041] Furthermore, the region setting unit 213 may set multiple determination regions such that the size of each determination region increases as the vehicle moves towards the front in the direction of travel. This is because the front of the vehicle in the direction of travel has more parallax information in the parallax image and the parallax image is often clearer than the rear of the vehicle in the direction of travel.

[0042] Here, with reference to Figure 4, an example of a method for identifying road boundary objects in front of the vehicle 1 when the area setting unit 213 sets the determination area will be explained.

[0043] The region setting unit 213 may associate a plurality of candidate points corresponding to road boundary objects in front of the vehicle 1 with a two-dimensional plane that intersects the vehicle height direction of the vehicle 1, based on the disparity image generated by the image generation unit 212. Alternatively, the region setting unit 213 may calculate a quadratic curve by quadratic approximation of the plurality of candidate points thus associated. Furthermore, the region setting unit 213 may set the points on the quadratic curve, when the quadratic curve calculated in this way is associated with the disparity image, as points corresponding to road boundary objects.

[0044] Specifically, the region setting unit 213 applies the principle of triangulation to the disparity image generated by the image generation unit 212. This allows the region setting unit 213 to identify, for example, objects in front of the vehicle 1 whose height along the vehicle height direction is equal to or greater than a reference height, and which extend for a reference length or longer in the depth direction of the stereo image, as road boundary objects. The region setting unit 213 may also associate multiple candidate points corresponding to road boundary objects in front of the vehicle 1 onto the XZ plane. Furthermore, the region setting unit 213 may calculate a quadratic curve by applying a known or arbitrary fitting method, such as a quadratic approximation using the least squares method, to the multiple candidate points associated on the XZ plane. Note that the vehicle width direction of the vehicle 1 is the X-axis, the vehicle height direction is the Y-axis, and the vehicle length direction is the Z-axis.

[0045] In the example shown in Figure 4, quadratic curve C represents a candidate curb on the left side in the direction of travel of vehicle 1. Ｌ Of the above, the determination region T is centered on a point 9m in front of vehicle 1 (Z=9m). Ｌ 1 and a determination area T centered on a point 13m in front of vehicle 1 (Z=13m). Ｌ 2 and the determination region T centered on a point 21m in front of vehicle 1 (Z=21m). Ｌ 3 is set in the two-dimensional parallax image. Also, as you move towards the front in the direction of travel of vehicle 1, the determination area T Ｌ 3. Judgment area T Ｌ2. Determination Region T Ｌ 1 is set to increase in size in this order. Note that the determination region T Ｌ 1 is, for example, 72 pixels (i) × 32 pixels (j), and the determination region T Ｌ 2 is, for example, 64 pixels (i) × 24 pixels (j), and the determination region T Ｌ 3 is, for example, 56 pixels (i) × 16 pixels (j).

[0046] Also, in the example shown in FIG. 4, the quadratic curve C representing the curb candidate on the right side of the traveling direction of the vehicle 1 Ｒ Among them, the determination region T centered on the point 9 m (Z = 9 m) in front of the vehicle 1 Ｒ 1, the determination region T centered on the point 13 m (Z = 13 m) in front of the vehicle 1 Ｒ 2, and the determination region T centered on the point 21 m (Z = 21 m) in front of the vehicle 1 Ｒ 3 are set in the two-dimensional parallax image. Also, as going toward the front side in the traveling direction of the vehicle 1, the determination region T Ｒ 3. Determination Region T Ｒ 2. Determination Region T Ｒ 1 is set to increase in size in this order. Note that the determination region T Ｒ 1 is, for example, 72 pixels (i) × 32 pixels (j), and the determination region T Ｒ 2 is, for example, 64 pixels (i) × 24 pixels (j), and the determination region T Ｒ 3 is, for example, 56 pixels (i) × 16 pixels (j).

[0047] (Ratio calculation unit) The ratio calculation unit 214 calculates the effective parallax ratio of the number of effective pixel blocks with effective parallax among all pixel blocks with respect to the total number of pixel blocks included in the determination region set by the region setting unit 213. Specifically, the ratio calculation unit 214 may calculate the total number of pixel blocks and the number of effective pixel blocks included in the determination region set by the region setting unit 213, and calculate the above-described effective parallax ratio by dividing the calculated number of effective pixel blocks by the total number of pixel blocks. Note that the ratio calculation unit 214 may store the calculated effective parallax ratio in association with the acquisition time of the stereo image in the storage unit 22.

[0048] If multiple determination areas are set, the ratio calculation unit 214 may calculate the effective disparity ratio of the number of effective pixel blocks to the total number of pixel blocks included in the multiple determination areas. In the example shown in Figure 4, the ratio calculation unit 214 calculates the determination area T Ｌ 1. Judgment area T Ｌ 2 and judgment area T Ｌ The number of pixels in all three that contain a 4x4 pixel block may be calculated as the total number of pixel blocks. Also, the ratio calculation unit 214 determines the determination area T Ｌ 1. Judgment area T Ｌ 2 and judgment area T Ｌ The number of 4x4 pixel blocks included in all of 3 in which parallax is effective may be calculated as the effective pixel block number. The ratio calculation unit 214 may calculate the left effective parallax ratio as an example of the effective parallax ratio described above by dividing the total number of pixel blocks calculated in this way by the effective pixel block number. Similarly, the ratio calculation unit 214 determines the determination area T Ｒ 1. Judgment area T Ｒ 2 and judgment area T Ｒ The number of pixels in all three that contain a 4x4 pixel block may be calculated as the total number of pixel blocks. Also, the ratio calculation unit 214 determines the determination area T Ｒ 1. Judgment area T Ｒ 2 and judgment area T Ｒ The number of 4x4 pixel blocks included in all of 3 in which parallax is effective may be calculated as the effective pixel block number. The ratio calculation unit 214 may calculate the right effective parallax ratio as an example of the effective parallax ratio described above by dividing the total number of pixel blocks calculated in this way by the effective pixel block number.

[0049] The ratio calculation unit 214 may independently calculate the effective disparity ratio for each of the multiple determination areas. In this case, the ratio calculation unit 214 can also select from among the multiple determination areas which determination areas will be used to calculate the effective disparity ratio for determination by the determination unit 215 described later, depending on the surrounding environment of the vehicle 1 that can be detected via the stereo camera 10 of the vehicle 1. For example, when the vehicle 1 is traveling on the left side (or right side) of the road, the determination area T shown in Figure 4 can be seen from the vehicle 1.Ｒ 1 (or judgment area T Ｌ 1) If the distance along the vehicle width direction exceeds the standard value, judgment area T Ｒ 1 (or judgment area T Ｌ 1) In some cases, a portion of the stereo image may not be included. In this case, the ratio calculation unit 214 determines the determination area T Ｒ 1 (or judgment area T Ｌ 1) may be excluded from the calculation of the effective disparity ratio. Similarly, when vehicle 1 is traveling on a curve, the determination area T shown in Figure 4 is... Ｒ 1 (or judgment area T Ｌ 1) If a portion of the stereo image is not included, the ratio calculation unit 214 determines the determination area T Ｒ 1 (or judgment area T Ｌ 1) may be excluded from the calculation of the effective disparity ratio.

[0050] (Determination Unit) The determination unit 215 determines the reliability of object recognition in front of the vehicle 1 based on the effective parallax ratio calculated by the ratio calculation unit 214. Specifically, if the effective parallax ratio calculated by the ratio calculation unit 214 is less than a threshold, the determination unit 215 may determine that the reliability of object recognition in front of the vehicle 1 is low. On the other hand, if the effective parallax ratio calculated by the ratio calculation unit 214 is not less than a threshold, the determination unit 215 may determine that the reliability of object recognition in front of the vehicle 1 is high. The threshold is, for example, 40%, but this disclosure is not limited thereto, and may be appropriately changed depending on the visibility impairment situation that can be recognized by the stereo camera 10 of the vehicle 1.

[0051] If the ratio calculation unit 214 has calculated the left effective parallax ratio and the right effective parallax ratio, respectively, the determination unit 215 may determine the reliability of object recognition separately for the left and right sides. That is, if the left effective parallax ratio is below a threshold, the determination unit 215 may determine that the reliability of object recognition on the left front of the vehicle 1 is low. On the other hand, if the right effective parallax ratio is below a threshold, the determination unit 215 may determine that the reliability of object recognition on the right front of the vehicle 1 is low. For example, if the left effective parallax ratio is 50% and the right effective parallax ratio is 35%, only the right side may be subjected to the HALT determination described later, and steering avoidance control for the right curb or side wall may be prohibited. In this case, steering avoidance control may be performed when the vehicle 1 approaches the left curb or side wall.

[0052] The determination unit 215 may determine whether or not to stop recognizing an object based on the recognition confidence level described above. Specifically, if the determination unit 215 determines that the recognition confidence level is low based on the effective disparity ratio calculated by the ratio calculation unit 214, it may decide to stop recognizing an object (hereinafter sometimes referred to as "HALT determination"). The determination unit 215 may also perform a HALT determination without going through the calculation of the recognition confidence level described above. That is, the determination unit 215 may directly determine to perform a HALT determination if the effective disparity ratio calculated by the ratio calculation unit 214 is less than a threshold, or it may directly determine not to perform a HALT determination if the effective disparity ratio calculated by the ratio calculation unit 214 is not less than a threshold.

[0053] (2-3. Example of Operation of the Judgment Device) Referring to Figure 5, an example of the operation of the judgment device 20 according to this embodiment will be explained in accordance with the flowchart.

[0054] This example of operation is typically performed when the driver assistance system installed in vehicle 1 is operating. The following aims to explain the overall flow of this example of operation; for detailed processing of each step, please refer to the description of the functional configuration of processing unit 21.

[0055] In step S11, the image acquisition unit 211 of the processing unit 21 acquires a stereo image including an object in front of the vehicle 1 from the stereo camera 10. The process then proceeds to step S12.

[0056] In step S12, the image generation unit 212 of the processing unit 21 generates a parallax image showing the parallax to an object in front of the vehicle 1, based on the stereo image acquired in step S11. In this example, the image generation unit 212 generates the parallax image shown in Figure 3. The process then proceeds to step S13.

[0057] In step S13, the region setting unit 213 of the processing unit 21 sets a determination region in the disparity image generated in step S12 that includes a point corresponding to the road boundary object in front of the vehicle 1. In this example, the region setting unit 213 sets a quadratic curve C corresponding to the snow wall that corresponds to the road boundary object on the left side in the direction of travel of the vehicle 1. Ｌ Determination regions T, which are separated from each other along the lines. Ｌ 1. Judgment area T Ｌ 2 and judgment area T Ｌ Set 3 as shown in Figure 4. Also, the area setting unit 213 is set to a quadratic curve C corresponding to the snow wall that corresponds to the road boundary object on the right side in the direction of travel of the vehicle 1. Ｒ Determination regions T, which are separated from each other along the lines. Ｒ 1. Judgment area T Ｒ 2 and judgment area T Ｒ Set 3 as shown in Figure 4. Then the process proceeds to step S14.

[0058] In step S14, the ratio calculation unit 214 of the processing unit 21 calculates the effective disparity ratio, which is the ratio of the number of effective pixel blocks to the total number of pixel blocks included in the determination region set in step S13. In this example, the ratio calculation unit 214 calculates the effective disparity ratio in the determination region T shown in Figure 4. Ｌ 1. Judgment area T Ｌ 2 and judgment area T Ｌ Regarding point 3, the left effective disparity ratio is calculated using the method described above. The ratio calculation unit 214 also determines the determination area T shown in Figure 4. Ｒ 1. Judgment area T Ｒ 2 and judgment area T ＲRegarding point 3, the effective disparity ratio on the right side is calculated using the method described above. After that, the process proceeds to step S15.

[0059] In step S15, the determination unit 215 of the processing unit 21 determines whether the effective disparity ratio calculated in step S14 is less than a threshold. In this example, as described above, the determination unit 215 uses the left effective disparity ratio and the right effective disparity ratio, respectively, to determine separately whether the effective disparity ratio is less than a threshold for the left and right sides. If it is determined that the effective disparity ratio is less than a threshold (step S15: Y), the process proceeds to step S16. On the other hand, if it is not determined that the effective disparity ratio is less than a threshold (step S15: N), the process ends. That is, the recognition of objects in front of the vehicle 1 by the driver assistance system continues.

[0060] In step S16, the determination unit 217 of the processing unit 21 determines that the reliability of recognizing an object in front of the vehicle 1 is low. In this example, if the determination unit 217 determines that the state in which the reliability of recognizing an object in front of the vehicle 1 is low continues for a predetermined time or longer, it may decide to stop recognizing the object in front of the vehicle 1 (i.e., make a HALT determination). The process then ends.

[0061] Furthermore, if a HALT (Hazard Attention) is detected, the processing unit 21, via the control device 7, controls the driver assistance system to stop recognizing the object. At this time, the processing unit 21 may also notify the driver of vehicle 1 via the HMI 13 that the driver assistance system has stopped recognizing the object. In addition, the processing unit 21, via the control device 7, may control the vehicle 1 to automatically move to a safe place such as the shoulder of the road using known or arbitrary driver assistance technology.

[0062] According to this example, the reliability of object recognition is determined using the effective parallax ratio of multiple determination areas set on the curb candidate in the parallax image, as shown in Figure 4, for example. Therefore, by using this recognition reliability, the driver assistance system can stop detecting curbs if, for example, the effective parallax ratio around the location where the stereo camera 10 is detecting a curb becomes small due to poor visibility. Thus, even in environments with poor visibility, misrecognition of road boundary objects by the stereo camera 10 can be suppressed.

[0063] However, this example of operation is just one example of how the determination result by the determination device 20 according to this embodiment can be applied, and this disclosure is not limited thereto. It can be applied to various processes for the driver assistance system to recognize an object in front of the vehicle 1 by stereo matching.

[0064] While preferred embodiments of this disclosure have been described in detail above with reference to the attached drawings, this disclosure is not limited to such examples. It is clear to any person with ordinary skill in the art to which this disclosure pertains that various modifications or alterations can be conceived within the scope of the technical idea set forth in the claims, and these too are understood to fall within the technical scope of this disclosure. For example, the functions, etc., included in each component or step can be rearranged in a logically consistent manner, and multiple components or steps can be combined into one or divided into separate components.

[0065] For example, the technology of this disclosure can be realized as a vehicle 1 equipped with the determination device 20 according to the above-described embodiment, and can also be realized as a determination method executed by a computer that can be configured in the same way as the determination device 20 according to the above-described embodiment. Furthermore, the technology of this disclosure can be realized as a determination program as a computer program that makes a computer that can be configured in the same way as the determination device 20 according to the above-described embodiment function, and can also be realized as a non-temporary tangible recording medium that records the determination program as a computer program.

[0066] 1: Vehicle, 10: Stereo camera, 20: Judgment device, 21: Processing unit, 211: Image acquisition unit, 212: Image generation unit, 213: Region setting unit, 214: Ratio calculation unit, 215: Judgment unit, 22: Storage unit

Claims

1. A determination device for determining the reliability of recognition of an object in front of a vehicle, comprising one or more processors and one or more memories connected to the one or more processors, wherein the one or more processors acquire a stereo image including the object in front, generate a parallax image showing the parallax to the object based on the stereo image, set a determination region in the parallax image including a point corresponding to a road boundary object in front, and determine the reliability of recognition based on the effective parallax ratio of the number of effective pixel blocks in which the parallax is effective out of the total number of pixel blocks to the total number of pixel blocks included in the determination region.

2. The determination device according to claim 1, wherein one or more processors set a plurality of determination regions as the determination region, each corresponding to a plurality of points separated from each other along the road boundary object on at least one of the left side and the right side in the direction of travel of the vehicle, and determine the recognition reliability based on the effective parallax ratio of the number of effective pixel blocks in which the parallax is effective out of the total number of pixel blocks to the total number of pixel blocks included in the plurality of determination regions.

3. The determination device according to claim 2, wherein one or more processors set the plurality of determination regions such that the size of the plurality of determination regions increases as they move toward the front side in the direction of travel of the vehicle.

4. The determination device according to any one of claims 1 to 3, wherein one or more processors associate a plurality of candidate points corresponding to the road boundary object with a two-dimensional plane intersecting in the vehicle height direction based on the disparity image, calculate a quadratic curve by quadratic approximation of the plurality of candidate points, and set as the point corresponding to the road boundary object a point on the quadratic curve when the quadratic curve is associated with the disparity image.

5. A determination method for determining the reliability of recognition of an object in front of a vehicle, comprising: a computer acquiring a stereo image including the object in front of the vehicle; generating a parallax image showing the parallax to the object based on the stereo image; setting a determination region in the parallax image that includes points corresponding to road boundary objects in front of the vehicle; and determining the reliability of recognition based on the effective parallax ratio of the number of effective pixel blocks in which the parallax is effective out of the total number of pixel blocks to the total number of pixel blocks included in the determination region.

6. A non-temporary tangible recording medium on which a determination program for determining the reliability of recognition of an object in front of a vehicle is recorded, wherein the determination program is recorded which causes a computer to: acquire a stereo image including the object in front of the vehicle; generate a parallax image showing the parallax to the object based on the stereo image; set a determination region in the parallax image including a point corresponding to a road boundary object in front of the vehicle; and determine the reliability of recognition based on the effective parallax ratio of the number of effective pixel blocks in which the parallax is effective out of the total number of pixel blocks to the total number of pixel blocks included in the determination region.

Images