Determination device, determination method, and recording medium

Abstract

A determination device according to the present invention for determining at least recognition reliability for road boundary objects in front of a vehicle acquires a stereo image including objects in front of the vehicle, identifies edge directions of edges included in the stereo image on the basis of a brightness change in the acquired stereo image, sets multiple regions along the multiple candidate points corresponding to the road boundary objects in the acquired stereo image, and determines the recognition reliability on the basis of the edge directions of the edges included in the set multiple regions from among the identified edge directions.

Description

Determination Device, Determination Method, and Recording Medium

[0001] The present disclosure relates to a determination device, a determination method, and a recording medium. In particular, the present disclosure relates to a determination device, a determination method, and a recording medium that at least determine the recognition reliability of road boundary objects in front of a vehicle.

[0002] Conventionally, in assisting a driver's driving operation of a vehicle, a technique for recognizing the surrounding environment of the vehicle is known.

[0003] For example, in Patent Document 1, a measurement unit that measures an object, an object detection unit that detects an object based on a signal acquired by the measurement unit, a road shape prediction unit that predicts the shape of a road on which the host vehicle travels, and a three-dimensional object selection unit that selects only three-dimensional objects within a predetermined range from the three-dimensional objects detected by the object detection unit and within the location of the road predicted by the road shape prediction unit, and a road shape estimation unit that estimates the shape of the road based on the position information of the three-dimensional objects selected by the three-dimensional object selection unit are provided, and a driving environment recognition device is disclosed.

[0004] Japanese Patent Application Laid-Open No. 2010-72973

[0005] The driving support system disclosed in Patent Document 1 performs driving support for a vehicle while recognizing three-dimensional objects within a predetermined range from a location on the road by means of a stereo camera.

[0006] By the way, for example, if there are obstacles such as vegetation or snow on a road boundary object such as a curb or a side wall, the edge directions of the edges included in the stereo image captured by the stereo camera are sparsely detected, and the recognition of the road boundary object may fall into an unstable state. In such a case, there is a risk that the road boundary object may be misdetected as a different object from the original.

[0007] In view of such circumstances, an object of the present disclosure is to provide a technique for suppressing the misdetection of a road boundary object as a different object from the original even when the recognition of the road boundary object using a stereo camera falls into an unstable state.

[0008] A determination device according to one embodiment of the present disclosure is a determination device for determining the reliability of recognition of a road boundary object in front of a vehicle, comprising one or more processors and one or more memories connected to the one or more processors, wherein the one or more processors acquire a stereo image including an object in front of the vehicle, identify the edge direction of an edge included in the stereo image based on the brightness change of the acquired stereo image, set a plurality of regions along a plurality of candidate points corresponding to the road boundary object in the acquired stereo image, and determine the reliability of recognition based on the edge direction of an edge included in the set plurality of regions from among the identified edge directions.

[0009] A determination method according to one embodiment of the present disclosure is a determination method for determining at least the reliability of the recognition of a road boundary object in front of a vehicle, comprising: a computer acquiring a stereo image including the object in front; identifying the edge direction of an edge included in the stereo image based on the brightness change of the acquired stereo image; setting a plurality of regions along a plurality of candidate points corresponding to the road boundary object in the acquired stereo image; and determining the reliability of the recognition based on the edge direction of an edge included in the set plurality of regions from among the identified edge directions.

[0010] A recording medium according to one embodiment of the present disclosure is a non-temporary tangible recording medium on which a determination program for determining the reliability of recognition of a road boundary object in front of a vehicle is recorded, wherein the recording medium contains a determination program that causes a computer to perform the following actions: to acquire a stereo image including the object in front of the vehicle; to identify the edge direction of an edge included in the stereo image based on the brightness change of the acquired stereo image; to set up a plurality of regions along a plurality of candidate points corresponding to the road boundary object in the acquired stereo image; and to determine the reliability of recognition based on the edge direction of an edge included in the set plurality of regions from among the identified edge directions.

[0011] According to one embodiment of the present disclosure, even if the recognition of road boundary objects using a stereo camera becomes unstable, it is possible to suppress the misdetection of road boundary objects as objects other than their actual nature.

[0012] This is a schematic diagram showing an example configuration of a vehicle equipped with a determination device according to one embodiment of the present disclosure. This is a block diagram showing an example configuration of a determination device according to one embodiment of the present disclosure. This is a diagram illustrating a stereo image and an edge image. This is a diagram illustrating the process of identifying the edge direction. This is a diagram illustrating the edge direction and the number of edges. This is a diagram illustrating the edge level. This is a diagram illustrating the mapping of multiple candidate points to the XZ plane. This is a diagram illustrating the mapping of multiple candidate points to the XZ plane. This is a diagram illustrating the mapping of multiple candidate points to the XY plane. This is a diagram illustrating the process of narrowing down multiple candidate points. This is a diagram illustrating the process of removing isolated points from multiple candidate points. This is a diagram illustrating the process of removing isolated points from multiple candidate points. This is a diagram illustrating multiple candidate points from which isolated points have been removed. This is a diagram illustrating the result of a first-order approximation on the XY plane. This is a diagram illustrating the result of a first-order approximation on the XY plane. This is a diagram illustrating that the shape of the frequency distribution differs depending on the type of road boundary object. This is a diagram illustrating the process of generating a frequency distribution from multiple candidate points. This is a diagram illustrating the degree of prominence of the frequency distribution. This is a diagram illustrating the result of a first-order approximation on the XY plane. This is a diagram illustrating the result of a first-order approximation on the XY plane. This is a flowchart illustrating a first example of operation of a determination device according to one embodiment of the present disclosure. This is a flowchart illustrating a second example of operation of a determination device according to one embodiment of the present disclosure. This is a diagram illustrating the determination criteria for HALT control.

[0013] Preferred embodiments of this disclosure will be described in detail below with reference to the attached drawings. In this specification and the drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant descriptions will be omitted.

[0014] (1. Vehicle) Referring to Figure 1, Vehicle 1 is configured as a four-wheeled automobile that transmits the drive torque output from the drive source 2 to the wheels. Vehicle 1 may be an automobile equipped with an internal combustion engine such as a gasoline engine or a diesel engine as the drive source 2, or it may be an electric vehicle equipped with a drive motor as the drive source 2. Examples of electric vehicles include BEV (Battery Electric Vehicle), HEV (Hybrid Electric Vehicle), PHEV (Plug-in Hybrid Electric Vehicle), or FCEV (Fuel Cell Electric Vehicle).

[0015] The drive source 2 outputs drive torque which is transmitted to the front wheel drive shaft 4F via a transmission (not shown) and a differential mechanism 3. The drive of the drive source 2 and the transmission is controlled by a control device 7, which will be described later. The combination of drive wheels and the driving method are not particularly limited, and the vehicle 1 may be a front-wheel drive vehicle, a rear-wheel drive vehicle, or a four-wheel drive vehicle. Furthermore, if the vehicle 1 is configured as an electric vehicle, it may be an electric vehicle equipped with a drive motor corresponding to each wheel.

[0016] Vehicle 1 is equipped with, in addition to the aforementioned drive force source 2, at least a steering device 5, a brake device 6, and a control device 7 as equipment used for driving control.

[0017] The steering device 5 is mounted on the front wheel drive shaft 4F. The steering device 5 may include an electric motor (not shown) and a gear mechanism (not shown), and adjusts the steering angle of the front wheels by being controlled by the control device 7.

[0018] The brake device 6 applies braking force to the wheels under the control of the control device 7. If the vehicle 1 is configured as an electric vehicle, the brake device 6 may be used in conjunction with regenerative braking provided by the drive motor, which serves as the driving force source 2.

[0019] The control device 7 includes at least one or more ECUs (Electronic Control Units) that control the driving of the power source 2, the steering device 5, and the brake device 6, respectively.

[0020] During manual operation, the control device 7 calculates the drive torque to be output to the drive source 2 based on the amount of operation of the accelerator pedal (not shown) by the driver. Also, during manual operation, the control device 7 calculates the braking force to be output to the brake device 6 based on the amount of operation of the brake pedal (not shown) by the driver. Also, during manual operation, the control device 7 controls the steering device 5 based on the steering angle of the steering wheel 8 by the driver.

[0021] During autonomous driving, the control device 7 controls the drive force source 2, steering device 5, and brake device 6, respectively, based on the target vehicle speed and target steering angle of the vehicle 1, which are appropriately set using known or arbitrary driver assistance technologies. However, the control device 7 is connected to a determination device 20, which will be described later, via a dedicated line or a communication means such as CAN (Controller Area Network) or LIN (Local Internet), and may perform control such as pulling the vehicle 1 to the side of the road according to the determination result of the determination device 20. Advanced Driver-Assistance Systems (ADAS) are examples of driver assistance technologies, but this disclosure is not limited thereto.

[0022] Vehicle 1 further includes a stereo camera 10, a vehicle state sensor 11, and a position detection sensor 12.

[0023] The stereo camera 10 includes a main camera 10a and a sub-camera 10b. The main camera 10a and the sub-camera 10b each include an image sensor such as a CCD (Charged Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor). The main camera 10a and the sub-camera 10b are each connected to the determination device 20 and, if necessary, to the control device 7 by wired or wireless communication means. The main camera 10a and the sub-camera 10b each capture the area in front of the vehicle 1 to generate a stereo image and transmit the generated stereo image to the determination device 20 and, if necessary, to the control device 7. Note that in Figure 1, the main camera 10a and the sub-camera 10b may be configured with their left and right sides reversed.

[0024] Specifically, the main camera 10a generates a reference image at a predetermined frame rate, and the sub-camera 10b generates a comparison image at a predetermined frame rate. As will be described in detail later, the determination device 20 acquires the reference image generated by the main camera 10a and the comparison image generated by the sub-camera 10b as stereo images. The determination device 20 generates a disparity image that can determine the distance to objects included in the stereo image by performing image processing such as stereo matching on the acquired stereo image.

[0025] In addition to the stereo camera 10, vehicle 1 may also be equipped with a camera (not shown) for imaging the area behind vehicle 1. Furthermore, vehicle 1 may also be equipped with one or more distance measuring sensors (not shown) from among radar sensors such as LiDAR (Light Detection And Ranging) or millimeter-wave radar and ultrasonic sensors.

[0026] The vehicle condition sensor 11 is a known or arbitrary sensor for detecting the state of the vehicle 1. For example, the vehicle condition sensor 11 may include a vehicle speed sensor for detecting the speed of the vehicle 1, and may include a steering angle sensor for detecting the steering angle of the steering wheel 8. The vehicle condition sensor 11 transmits the detection results to the control device 7 and, if necessary, to the determination device 20.

[0027] The position detection sensor 12 may include a GNSS (Global Navigation Satellite System) sensor that receives satellite signals from positioning satellites such as GPS (Global Positioning System) satellites. The position detection sensor 12 transmits information indicating the current position of the vehicle 1, which is included in the received satellite signals, to the control device 7, and to the determination device 20 as needed.

[0028] In addition, vehicle 1 may further include a known or arbitrary HMI (Human Machine Interface) 13 that presents various information to the driver of vehicle 1 through image or text display or audio output, etc.

[0029] (2. Determination Device) The determination device 20 according to this embodiment will be described in detail with reference to Figure 2.

[0030] (2-1. Configuration Example) The determination device 20 functions as a device that determines at least the reliability of recognizing road boundary objects in front of the vehicle 1 by having one or more CPUs (Central Processing Units) or other processors execute a computer program. The computer program is a computer program that causes the processor to execute the operations that the determination device 20 should perform, which will be described later. The computer program executed by the processor may be recorded on a recording medium that functions as a storage unit (memory) 22, which will be described later, or it may be recorded on a recording medium built into the determination device 20 or on any recording medium that can be attached externally to the determination device 20.

[0031] The recording medium for storing computer programs may include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs, DVDs, and Blu-ray®; magneto-optical media such as floppy disks; memory elements such as RAM (Random Access Memory) and ROM (Read Only Memory); flash memory such as USB (Universal Serial Bus) memory and SSD (Solid State Drive); and other media capable of storing programs.

[0032] The determination device 20 is connected to at least the control device 7 and the stereo camera 10. It is also possible that some or all of the components of the determination device 20 are provided on the control device 7 side, or that they are provided on the server (not shown) side capable of communicating with the vehicle 1.

[0033] The determination device 20 comprises at least a processing unit 21 and a storage unit 22. If some or all of the components of the determination device 20 are provided on the server side, the determination device 20 further comprises a communication unit (not shown) which includes a communication interface for communicating with the vehicle 1 via a known or arbitrary network (not shown).

[0034] (Processing Unit) The processing unit 21 comprises one or more processors such as CPUs and various peripheral components. Part or all of the processing unit 21 may consist of updatable components such as firmware, or it may be a program module that is executed by instructions from the CPU, etc.

[0035] (Storage Unit) The storage unit 22 is composed of one or more storage elements such as RAM or ROM that are connected to the processing unit 21 in a communicative manner. However, the type and number of storage units 22 are not particularly limited. The storage unit 22 stores information such as computer programs executed by the processing unit 21, various parameters used in arithmetic processing, detection results, and calculation results.

[0036] (2-2. Functional Configuration of the Processing Unit) The functional configuration of the processing unit 21 of the determination device 20 will be described below. The processing unit 21 comprises at least an image acquisition unit 211, an edge detection unit 212, and a determination unit 213. Each of the image acquisition unit 211, the edge detection unit 212, and the determination unit 213 is a function realized by the execution of a computer program by one or more processors such as a CPU. However, some or all of the image acquisition unit 211, the edge detection unit 212, and the determination unit 213 may be configured using analog circuits.

[0037] (Image Acquisition Unit) The image acquisition unit 211 acquires a stereo image including an object in front of the vehicle 1. Specifically, the image acquisition unit 211 may acquire a reference image including an object in front of the vehicle 1, captured by the main camera 10a included in the stereo camera 10, and a comparison image including an object in front of the vehicle 1, captured by the sub-camera 10b included in the stereo camera 10, as a stereo image. It is preferable that the image acquisition unit 211 acquires the stereo image at a calculation cycle synchronized with the frame rate of the stereo camera 10, and the acquired stereo image may be stored in the storage unit 22 linked to the acquisition time.

[0038] (Edge detection unit) The edge detection unit 212 identifies the edge direction of an edge included in a stereo image based on the brightness change of the stereo image including an object in front of the vehicle 1 acquired by the image acquisition unit 211. The object in front of the vehicle 1 may include road boundary objects in front of the vehicle 1. Examples of road boundary objects are not particularly limited, but include curbs, side walls, or snow walls.

[0039] Specifically, the edge detection unit 212 may arbitrarily extract a target cell from at least one stereo image, which is either a reference image captured by the main camera 10a or a comparison image captured by the sub-camera 10b. The edge detection unit 212 may also identify eight surrounding cells that surround the extracted target cell. Furthermore, the edge detection unit 212 may identify the edge direction of the edge in the target cell based on the amount of change from the brightness of the target cell to the brightness of each surrounding cell. In this case, the edge detection unit 212 may identify the direction of a vector as the edge direction, starting from the center of the target cell and ending at the center of the surrounding cell with the largest change in brightness among the eight surrounding cells. While a 1x1 pixel is given as an example of a target cell, this disclosure is not limited to this.

[0040] More specifically, the edge detection unit 212 may set edge direction 0 at the 12 o'clock position in the forward field of view from the vehicle 1, and set edge directions 1, 2, 3, 4, 5, 6, and 7 in this order at 45-degree intervals clockwise from edge direction 0. In this case, the edge detection unit 212 may identify which of edge directions 0 to 7 the edge in each cell included in the reference image corresponds to. In this way, the edge detection unit 212 can generate the edge image shown in the lower part of Figure 3 from, for example, the stereo image shown in the upper part of Figure 3. The edge image is an image in which edges having a predetermined edge direction are extracted from the stereo image, and the lower part of Figure 3 illustrates the case in which edge directions 3 and 7 are extracted from edge directions 0 to 7. For curbs or side walls, which are examples of road boundary objects, the edge direction is clearly detected, but when there is vegetation or snow on the curb or snow wall, the edge direction tends to be detected sparsely.

[0041] In addition to specifying the edge direction as described above, the edge detection unit 212 sets multiple regions along multiple candidate points corresponding to road boundary objects in front of the vehicle 1 in the stereo image acquired by the image acquisition unit 211.

[0042] Specifically, the edge detection unit 212 may set multiple regions at predetermined intervals along multiple candidate points corresponding to road boundary objects in the stereo image used to identify the edge direction among the reference image and comparison image. Here, the multiple candidate points corresponding to road boundary objects may also be multiple candidate points used by the correspondence unit 215, described later, to calculate a quadratic curve. The edge detection unit 212 may set the above-mentioned multiple regions such that the size of the multiple regions decreases as the vehicle moves towards the far side in the direction of travel. In the example shown in Figure 4, multiple regions are set centered on each of the multiple candidate points Pi (where i is a natural number) that are shown along the curb between the lane and the curb that demarcate the driving area of ​​the vehicle 1. Also in Figure 4, the size of the region at a point 8m in front of the vehicle 1 (Z=8) is shown as 21 pixels × 21 pixels as an example, enlarged with a thick dashed frame. However, the size of the region is not limited to this, and it is sufficient if it is the same size vertically and horizontally.

[0043] The edge detection unit 212 may calculate the total number of edges included in the plurality of regions set as described above. Further, the edge detection unit 212 may calculate the number of edges among the edges included in the plurality of regions set, where the edge direction is diagonally left and diagonally right with respect to the traveling direction of the vehicle 1. Further, the edge detection unit 212 may calculate the ratio of the number of edges to the total number of edges calculated. Specifically, the edge detection unit 212 may calculate, for each of the plurality of regions, for example, as shown in FIG. 5, the number of edges x0 in the edge direction 0, the number of edges x1 in the edge direction 1, the number of edges x2 in the edge direction 2, the number of edges x3 in the edge direction 3, the number of edges x4 in the edge direction 4, the number of edges x5 in the edge direction 5, the number of edges x6 in the edge direction 6, and the number of edges x7 in the edge direction 7. Note that x8 is the number of edge directions invalidated in the stereo image. Further, the edge detection unit 212 may calculate, for the edges included in the plurality of regions, for example, using the following formula (1), the ratio of the edges diagonally left with respect to the traveling direction of the vehicle 1 (hereinafter referred to as "left component ratio"). Further, the edge detection unit 212 may calculate, for the edges included in the plurality of regions, for example, using the following formula (2), the ratio of the edges diagonally right with respect to the traveling direction of the vehicle 1 (hereinafter referred to as "right component ratio").

[0044]

[0045]

[0046] Note that the edge detection unit 212 may calculate the left component ratio for each frame using the above formula (1) with respect to the stereo images of a predetermined number of frames acquired by the image acquisition unit 211, and arithmetically average the left component ratios calculated for each frame over the predetermined number of frames. Further, the edge detection unit 212 may calculate the right component ratio for each frame using the above formula (2) with respect to the stereo images of a predetermined number of frames acquired by the image acquisition unit 211, and arithmetically average the right component ratios calculated for each frame over the predetermined number of frames. Thereby, the left component ratio and the right component ratio can be stabilized respectively. The predetermined number of frames is, for example, 10 frames, but the present disclosure is not limited thereto.

[0047] Here, the mapping unit 215, described later, may interpolate some of the candidate points when identifying multiple candidate points corresponding to road boundary objects. "Interpolation" means, for example, when a gap is formed in a curb or side wall, which is an example of a road boundary object, the process of identifying multiple candidate points corresponding to the curb or side wall is assumed to exist continuously at the gap. In this case, when the edge detection unit 212 calculates the left component ratio and the right component ratio described above, it is preferable to exclude the edges included in the interpolated region from the determination of the recognition reliability, described later, among the edges included in the multiple regions. That is, the edge detection unit 212 does not add the number of edges included in the interpolated region among the edges included in the multiple regions in the numerator of equation (1) and equation (2) above.

[0048] (Determination Unit) The determination unit 213 determines the recognition reliability of the road boundary object based on the edge direction calculated by the edge detection unit 212. Here, the recognition reliability of the road boundary object may be determined separately for the left and right sides. Specifically, the determination unit 213 may determine that the recognition reliability of the road boundary object should be set to edge level LV0 if the left component ratio calculated by the edge detection unit 212 is equal to or greater than the first edge threshold. The determination unit 213 may also determine that the recognition reliability of the road boundary object should be set to edge level LV1 if the left component ratio is equal to or greater than the second edge threshold and less than the first edge threshold. The determination unit 213 may also determine that the recognition reliability of the road boundary object should be set to edge level LV2 if the left component ratio is less than the second edge threshold. On the other hand, the determination unit 213 may also determine that the recognition reliability of the road boundary object should be set to edge level LV0 if the right component ratio calculated by the edge detection unit 212 is equal to or greater than the first edge threshold. Furthermore, the determination unit 213 may determine that the road boundary object recognition reliability should be set to edge level LV1 if the proportion of the right component is greater than or equal to the second edge threshold and less than the first edge threshold. Alternatively, the determination unit 213 may determine that the road boundary object recognition reliability should be set to edge level LV2 if the proportion of the right component is less than the second edge threshold. Note that the second edge threshold is smaller than the first edge threshold.

[0049] For example, Figure 6 illustrates that the edge level differs depending on the type of road boundary object. Here, the first edge threshold is set to 30%, and the second edge threshold is set to 20%. Furthermore, the LV value of the edge level is set to increase as the recognition reliability of the road boundary object decreases. However, this disclosure is not limited to this, and for example, the LV value of the edge level can be set arbitrarily. Note that edge levels LV0 to LV2 are examples of the recognition reliability of road boundary objects and may be referred to as the "first recognition reliability of road boundary objects."

[0050] As will be described in more detail later, the determination unit 213 may decide whether or not to stop recognizing the road boundary object based on the recognition reliability of the road boundary object determined by the above-described process.

[0051] The basic configuration of the processing unit 21 of the determination device 20 has been described above, but the processing unit 21 may further include an image generation unit 214, a correspondence unit 215, a variation calculation unit 216, a preprocessing unit 217, a slope correlation calculation unit 218, and a frequency distribution generation unit 219. Each of the image generation unit 214, correspondence unit 215, variation calculation unit 216, preprocessing unit 217, slope correlation calculation unit 218, and frequency distribution generation unit 219 is a function realized by the execution of a computer program by one or more processors such as a CPU. However, some or all of the image generation unit 214, correspondence unit 215, variation calculation unit 216, preprocessing unit 217, slope correlation calculation unit 218, and frequency distribution generation unit 219 may be configured using analog circuits.

[0052] (Image Generation Unit) The image generation unit 214 generates a parallax image capable of determining the distance to an object in front of the vehicle 1 based on the stereo image acquired by the image acquisition unit 211. Specifically, the image generation unit 214 applies known or arbitrary stereo matching processing to the reference image and comparison image, which are stereo images. More specifically, the image generation unit 214 searches the comparison image for blocks corresponding to blocks arbitrarily extracted from the reference image, and calculates the parallax of the blocks in the reference image from the positional relationship between the blocks in the reference image and the blocks in the comparison image. It is preferable that the image generation unit 214 generates a parallax image by performing this series of processing on all blocks included in the reference image. It is preferable that the image generation unit 214 generates a parallax image each time the image acquisition unit 211 acquires a stereo image, and the generated parallax image may be stored in the storage unit 22 in association with the acquisition time of the stereo image. The value or color of each block in the parallax image indicates the parallax and corresponds to the distance to the block in three-dimensional real space corresponding to each block. Furthermore, while each block is exemplified as a 4x4 pixel configuration, this disclosure is not limited to this and can be set arbitrarily.

[0053] (Mapping unit: XZ plane) The mapping unit 215 identifies a plurality of candidate points corresponding to an object in front of the vehicle 1 based on the disparity image generated by the image generation unit 214, and maps the identified plurality of candidate points to a two-dimensional plane that intersects in the vehicle height direction of the vehicle 1.

[0054] Specifically, the coordinates (i, j) in the disparity image and the disparity of the block to which the coordinates (i, j) belong are mapped one-to-one with the coordinates (x, y, z) of each block in a three-dimensional real space where the vehicle width direction of the vehicle 1 is the X-axis, the vehicle height direction is the Y-axis, and the vehicle length direction is the Z-axis, by coordinate transformation based on the principle of triangulation. The mapping unit 215 then applies the principle of triangulation to the disparity image generated by the image generation unit 214. As a result, the mapping unit 215 may map multiple candidate points corresponding to objects including road boundary objects in front of the vehicle 1 onto the XZ plane, for example, as shown by a point cloud consisting of multiple candidate points in Figures 7 and 8, respectively. Preferably, the mapping unit 215 maps multiple candidate points onto the XZ plane each time the image generation unit 214 generates a disparity image, and the result of the mapping onto the XZ plane may be stored in the storage unit 22 in association with the acquisition time of the stereo image.

[0055] (Mapping Unit: XY Plane) The mapping unit 215 may identify a plurality of candidate points corresponding to objects in front of the vehicle 1 based on the disparity image generated by the image generation unit 214, and map the identified plurality of candidate points to a further two-dimensional plane that intersects in the vehicle length direction of the vehicle 1. Specifically, the mapping unit 215 may apply the principle of triangulation to the disparity image generated by the image generation unit 214 to map a plurality of candidate points corresponding to objects including road boundary objects in front of the vehicle 1 to the XY plane, for example, as shown in Figure 9. It is preferable that the mapping unit 215 maps a plurality of candidate points to the XY plane at predetermined intervals (e.g., 4 m) along the Z axis each time the image generation unit 214 generates a disparity image, and the result of the mapping to the XY plane may be stored in the storage unit 22 in association with the acquisition time of the stereo image.

[0056] In the example shown in Figure 9, multiple candidate points corresponding to an object including a road boundary object at a point 8 m (Z = 8 m) in front of vehicle 1 are mapped onto the XY plane. In Figure 9, the point cloud consisting of multiple candidate points located along the X-axis from near X = -1.0 m to near X = 5.0 m corresponds to the road surface on which vehicle 1 is traveling. Also in Figure 9, the point cloud consisting of multiple candidate points located at an incline rising from the road surface from near X = -1.0 m to near X = -2.5 m corresponds to a snow wall as a road boundary object.

[0057] (Variation Calculation Unit) The variation calculation unit 216 may calculate a quadratic curve by quadratic approximation of a plurality of candidate points corresponding to road boundary objects that the correspondence unit 215 has mapped onto a two-dimensional plane. Alternatively, the variation calculation unit 216 may calculate the variation of the plurality of candidate points corresponding to road boundary objects mapped by the correspondence unit 215 from the calculated quadratic curve.

[0058] Specifically, the variation calculation unit 216 may perform a quadratic approximation using the least squares method or the like on a plurality of candidate points corresponding to road boundary objects on the XZ plane, for example, as shown in Figure 7. This allows the variation calculation unit 216 to calculate a quadratic curve formed by fitting a point cloud consisting of a plurality of candidate points corresponding to road boundary objects. More specifically, when curbs or side walls corresponding to road boundary objects are converted to distance using a disparity image generated by the image generation unit 214, and candidate points are plotted on the XZ plane at intervals of, for example, 100 mm in the Z direction and 50 mm in the X direction, there is a tendency for concentrated areas to occur where the candidate points are clustered together. Furthermore, the height positions of candidate points moving from slightly inside to outside the concentrated areas tend to show a difference in elevation at the boundary between the road surface and the curb or side wall. Therefore, the variation calculation unit 216 may retain candidate points near such elevation differences as candidate points for curbs or side walls and calculate a quadratic curve. If there are multiple concentrated areas, the concentrated areas further inside may be used as candidates. In Figure 7, multiple candidate points corresponding to road boundary objects up to a point 60m in front of vehicle 1 (Z=60) are fitted on the XZ plane using quadratic curves. However, the fitting range is not necessarily limited to a point 60m in front of vehicle 1 and can be set as appropriate.

[0059] Furthermore, the variation calculation unit 216 may calculate the variation in the amount of deviation along the X-axis, which corresponds to the vehicle width direction of the vehicle 1, from the quadratic curve, as the variation of multiple candidate points corresponding to road boundary objects on the XZ plane from the quadratic curve. Note that "variation" may also be the standard deviation, which can be calculated as the positive square root of the mean square of the amount of deviation. In Figure 7, the variation when the road boundary object is a side wall, where the behavior of multiple candidate points tends to be stable, is illustrated, and the standard deviation is 121. On the other hand, in Figure 8, the variation when the road boundary object is a snow wall, where the behavior of multiple candidate points tends to be unstable, is illustrated, and the standard deviation is 296. In this way, road boundary objects can be identified by comparing the raw values ​​of multiple candidate points on the XZ plane with the quadratic curve.

[0060] The variation from the quadratic curve calculated in this way can be used by the determination unit 213 to determine the reliability of road boundary object recognition. That is, the determination unit 213 may determine the reliability of road boundary object recognition based on the variation from the quadratic curve calculated by the variation calculation unit 216. Specifically, the determination unit 213 may determine that the reliability of road boundary object recognition is lower when the variation of the deviation amount calculated by the variation calculation unit 216 is greater than a standard value, compared to when it is less than the standard value. More specifically, the determination unit 213 may determine to set the reliability of road boundary object recognition to variation level LV2 if the standard deviation from the quadratic curve is greater than or equal to the first variation threshold. Alternatively, the determination unit 213 may determine to set the reliability of road boundary object recognition to variation level LV1 if the standard deviation from the quadratic curve is greater than or equal to the second variation threshold and less than the first variation threshold. Furthermore, the determination unit 213 may determine that the recognition reliability of road boundary objects should be set to variation level LV0 if the standard deviation from the quadratic curve is less than the second variation threshold. Note that the second variation threshold is smaller than the first variation threshold.

[0061] The variation levels LV0 to LV2 are examples of the recognition reliability of road boundary objects and may be referred to as the "second recognition reliability of road boundary objects." The LV value of the variation level is set so that it increases as the recognition reliability of the road boundary object decreases. However, this disclosure is not limited to this, and for example, the LV value of the variation level can be set arbitrarily.

[0062] (Pre-processing unit) From the viewpoint of reducing processing time and processing load, the pre-processing unit 217 may perform the following narrowing down as pre-processing on the plurality of candidate points associated by the correspondence unit 215. That is, the pre-processing unit 217 may narrow down the candidate points that are included in a predetermined calculation range from among the plurality of candidate points that correspond to objects including road boundary objects in front of the vehicle 1, which have been associated by the correspondence unit 215 on a further two-dimensional plane. Specifically, the pre-processing unit 217 may narrow down the candidate points that are included in a predetermined calculation range that excludes the vicinity of the road surface and includes the vicinity of road boundary objects, in an XY plane corresponding to a further two-dimensional plane, for example, as shown in the shaded area of ​​Figure 10.

[0063] The area near the road surface is not particularly limited, but examples include a range within 30 mm along the positive direction of the Y-axis from the Y-coordinate position of the road surface that can be identified from multiple candidate points on the XY plane by known or arbitrary fitting techniques for determining the shape of the road surface. The area near the road boundary object is not particularly limited, but examples include the X-coordinate position of the road boundary object X ０ An example is a range within 500 mm along the positive direction of the X-axis and within 800 mm along the negative direction of the X-axis. Here, the preprocessing unit 217 determines the intersection point of the quadratic curve of a plurality of candidate points corresponding to road boundary objects mapped onto the XZ plane by the method described above and the XY plane, and sets the X coordinate position X ０ This may also be done. For example, in Figure 10, multiple candidate points corresponding to an object including a road boundary object at a point 8 m (Z = 8 m) in front of vehicle 1 are mapped on the XY plane, and a shaded area is identified as a predetermined calculation range. Note that the road boundary object may be a curb, or if there is no curb, it may be an object with a continuous height in the depth direction, such as a side wall or snow wall, and if such an object does not exist, the HALT determination itself described later may not be performed, and the result of the previous HALT determination may be retained.

[0064] The preprocessing unit 217 may further perform a process to remove isolated points from candidate points included in a predetermined calculation range, from the viewpoint of improving the accuracy of the first-order approximation without increasing the processing time. That is, the preprocessing unit 217 may further divide the predetermined calculation range into multiple cells, identify isolated points from among the candidate points included in the predetermined calculation range based on the number of candidate points included in each cell, and perform a preprocessing step to remove the identified isolated points. This method is simpler and can shorten the processing time compared to median filters or Gaussian filters, which are commonly used to reduce noise in images.

[0065] Specifically, the preprocessing unit 217 may further divide the predetermined calculation range into multiple cells by setting a rectangular cell, for example, as shown in Figure 11, where the length of the side along the X-axis is 50 mm and the length of the side along the Y-axis is 10 mm. The preprocessing unit 217 may also calculate the sum of candidate points contained in a total of nine cells: a target cell C arbitrarily extracted from the predetermined calculation range and eight surrounding cells C' that surround the target cell C. Furthermore, if the sum of candidate points is less than a deletion threshold, the preprocessing unit 217 may identify the candidate points contained in the target cell C as isolated points and delete the identified isolated points.

[0066] In the example shown in Figure 11, the deletion threshold is 3, and the sum of the candidate points in target cell C1 and surrounding cell C'1 is 3, so the candidate points in target cell C1 are not deleted. On the other hand, the sum of the candidate points in target cell C2 and surrounding cell C'2 is 2, so the candidate points in target cell C2 are deleted. Similarly, the sum of the candidate points in target cell C3 and surrounding cell C'3 is 2, so the candidate points in target cell C3 are deleted. It is preferable that the preprocessing unit 217 treats all cells included in a predetermined calculation range as target cells and performs this preprocessing on each target cell. As a result, for example, candidate points included in the solid thick frame within the predetermined calculation range shown by the dashed thick frame in Figure 12 are identified as isolated points and deleted, for example, as shown in Figure 13.

[0067] (Slope Correlation Calculation Unit) The slope correlation calculation unit 218 may calculate the slope a and correlation coefficient r of the linear approximation formula by linearly approximating a plurality of candidate points that are included in a predetermined calculation range and from which isolated points have been removed. However, the slope correlation calculation unit 218 may perform linear approximation only if the number of candidate points is equal to or greater than a reference value appropriately set to avoid false detection, and may not perform linear approximation if it is less than the reference value.

[0068] Specifically, the slope correlation calculation unit 218 may calculate the slope a of the linear approximation formula, for example, represented by equation (3) below, by linearly approximating a plurality of candidate points that are included in a predetermined calculation range and do not include isolated points on the XY plane.

[0069]

[0070] Furthermore, the slope correlation calculation unit 218 may calculate a correlation coefficient r, which indicates the strength of the linear relationship between the X-coordinate position (lateral position) on the X-axis corresponding to the vehicle width direction and the Y-coordinate position (vertical position) on the Y-axis corresponding to the vehicle height direction, for a plurality of candidate points that are included in a predetermined calculation range and from which isolated points have been removed, using, for example, the following equation (4).

[0071]

[0072] As a result, for example, as shown in Figure 14, a snow wall, which is an example of a road boundary structure, tends to have an absolute value of the correlation coefficient r approaching 1.0 because the point cloud consisting of multiple candidate points is inclined with respect to both the X and Y axes. On the other hand, as shown in Figure 15, a curb, which is another example of a road boundary structure, tends to have an absolute value of the correlation coefficient r approaching 0.0 because the point cloud consisting of multiple candidate points does not easily follow the linear approximation formula. Therefore, road boundary structures can be identified by using the correlation coefficient r.

[0073] In this embodiment, the correlation coefficient r for the road boundary object on the front left side of vehicle 1 is defined to take a value from 0.0 to -1.0, with the value approaching -1.0 as the correlation is higher. On the other hand, the correlation coefficient r for the road boundary object on the front right side of vehicle 1 is defined to take a value from 0.0 to 1.0, with the value approaching 1.0 as the correlation is higher. However, these definitions do not unduly limit this disclosure and can be appropriately changed according to the specifications of the driver assistance system, including whether the coefficient is positive or negative.

[0074] The slope a and correlation coefficient r calculated in this manner can be used by the determination unit 213 to determine the reliability of road boundary object recognition. That is, the determination unit 213 may determine the reliability of road boundary object recognition based on at least one of the slope a and correlation coefficient r calculated by the slope correlation calculation unit 218.

[0075] Specifically, the determination unit 213 may determine that the recognition reliability of the road boundary object is set to "with slope correlation" if the slope a is greater than or equal to the slope threshold and the correlation coefficient r is greater than or equal to the correlation threshold. On the other hand, the determination unit 213 may determine that the recognition reliability of the road boundary object is set to "without slope correlation" if the slope a is less than the slope threshold or the correlation coefficient r is less than the correlation threshold. In other words, "with slope correlation" and "without slope correlation" are examples of the recognition reliability of the road boundary object and may be referred to as the "third recognition reliability of the road boundary object". Here, "with slope correlation" means that the recognition reliability of the road boundary object is lower than "without slope correlation". The slope threshold is, for example, 0.2 and the correlation threshold is, for example, 0.7, but this disclosure is not limited to these and can be set as appropriate.

[0076] (Frequency Distribution Generation Unit) When the determination unit 213 determines the reliability of road boundary object recognition, the frequency distribution generation unit 219 may perform the following processing from the viewpoint of improving the recognition accuracy of road boundary objects. That is, the frequency distribution generation unit 219 may generate a frequency distribution (e.g., a histogram) that shows the number of candidate points included in a predetermined interval among a plurality of candidate points at predetermined intervals along the vehicle width direction of the vehicle 1 on a further two-dimensional plane intersecting the vehicle length direction of the vehicle 1.

[0077] This is based on the finding that the shape of the frequency distribution generated from multiple candidate points corresponding to road boundary objects differs depending on the type of road boundary object, as shown in Figure 16, for example. As will be explained in more detail later, the frequency distributions of patterns 1, 2, and 3 have frequencies that stand out significantly from the mean of the frequency distribution compared to the frequency distributions of patterns 4 and 5, indicating that the road boundary object is likely to be a curb. On the other hand, the frequency distributions of patterns 4 and 5 do not have frequencies that stand out significantly from the mean of the frequency distribution, indicating that the road boundary object is less likely to be a curb and more likely to be a side wall or snow wall.

[0078] Specifically, the frequency distribution generation unit 219 may divide the X-axis into grid-like sections at intervals of, for example, 50 mm and the Y-axis into sections at intervals of, for example, 10 mm, as shown in Figure 17. Alternatively, the frequency distribution generation unit 219 may generate a frequency distribution where each interval when the X-axis is divided into 50 mm intervals is used as a class, and the number of candidate points in each class is used as the frequency, as shown in Figure 18. In Figure 18, "160" is the class value calculated by dividing 5000 mm by 150 mm. The frequency distribution generation unit 219 may also calculate the mean and standard deviation of the generated frequency distribution using a known or arbitrary method. For the curb on the front left side of the vehicle 1, the range of the frequency distribution generation may be set to the range from -8000 mm to 0 mm on the X-axis and the range from -800 mm to 800 mm on the Y-axis, as shown in Figure 17. On the other hand, for the curb on the front right side of vehicle 1, the frequency distribution generation range may be set to the range from 0 mm to 8000 mm on the X-axis and from -800 mm to 800 mm on the Y-axis, as shown in Figure 17.

[0079] The frequency distribution generated in this manner can be used by the determination unit 213 to determine the reliability of road boundary object recognition. That is, the determination unit 213 may determine the reliability of road boundary object recognition based on the degree of prominence of the frequency distribution generated by the frequency distribution generation unit 219.

[0080] Specifically, the determination unit 213 may determine the reliability of road boundary object recognition based on whether or not there exists a frequency class whose degree of prominence in the frequency distribution is greater than or equal to a threshold set based on the mean and standard deviation of the frequency distribution. More specifically, the determination unit 213 may determine, for example, whether or not there exists a frequency that satisfies equation (5) below. Hereinafter, the frequency that satisfies equation (5) below may be referred to as the "first frequency class". Note that the right-hand side of equation (5) below is an indicator that 95% of the entire data falls within the frequency on the left-hand side of equation (5) below.

[0081]

[0082] If it is determined that a first height class exists, the determination unit 213 may determine the reliability of road boundary object recognition based on the difference between the maximum and minimum height positions of candidate points along the vehicle height direction of vehicle 1 in the class to which the first height class belongs. More specifically, the determination unit 213 may determine whether the difference between the maximum and minimum Y coordinate positions of candidate points in the class to which the first height class belongs is greater than or equal to a height threshold (e.g., 140 mm). If it is determined that the difference is greater than or equal to the height threshold, the determination unit 213 may determine to set the reliability of road boundary object recognition to curb level LV2. Here, the LV value of the curb level is set so that the higher the curb element, the stronger the reliability of road boundary object recognition. However, this disclosure is not limited thereto, and for example, the LV value of the curb level can be set arbitrarily.

[0083] If the difference is not determined to be equal to or greater than the height threshold, the determination unit 213 may decide to adopt the highest level among the curb level LV1, wall level LV2, and wall level LV1 described later as the reliability level for recognizing road boundary objects. Also, if it is not determined that a first height class exists, the determination unit 213 may determine, for example, whether or not there is a frequency that satisfies equation (6) below. Hereinafter, frequencies that satisfy equation (6) below may be referred to as the "second height class".

[0084]

[0085] If it is determined that a second height class exists, the determination unit 213 may determine the recognition reliability of road boundary objects based on the difference between the maximum and minimum height positions of candidate points along the vehicle height direction of vehicle 1 in the class to which the second height class belongs. More specifically, the determination unit 213 may determine whether the difference between the maximum and minimum Y coordinate positions of candidate points in the class to which the second height class belongs is greater than or equal to a height threshold (e.g., 140 mm). If it is determined that the difference is greater than or equal to the height threshold, the determination unit 213 may determine to set the recognition reliability of road boundary objects to curb level LV1. If it is not determined that the difference is greater than or equal to the height threshold, the determination unit 213 may determine to adopt the highest level among wall level LV2 and wall level LV1, described later, as the recognition reliability of road boundary objects. Furthermore, if it is not determined that a second height level exists, the determination unit 213 may decide to adopt the highest level among wall level LV2, wall level LV1, and wall level LV0, which will be described later, as the reliability level for recognizing road boundary objects.

[0086] In this way, for example, curbs of patterns 1 to 3 shown in Figure 16 can be distinguished. The first and second height classes are examples of height classes determined based on thresholds set based on the mean and standard deviation of the frequency distribution. Therefore, the number of height classes and the thresholds for determining the height classes in this disclosure are not limited to those described above.

[0087] On the other hand, the determination unit 213 may determine the recognition reliability of the road boundary object based on the inclination a calculated by the inclination correlation calculation unit 218. Specifically, the determination unit 213 may determine that the recognition reliability of the road boundary object should be set to wall level LV2 if the absolute value of inclination a is greater than or equal to the first inclination threshold. Alternatively, the determination unit 213 may determine that the recognition reliability of the road boundary object should be set to wall level LV1 if the absolute value of inclination a is greater than or equal to the second inclination threshold which is smaller than the first inclination threshold, and less than the first inclination threshold. Furthermore, the determination unit 213 may determine that the recognition reliability of the road boundary object should be set to wall level LV0 if the absolute value of inclination a is less than the second inclination threshold.

[0088] For example, Figure 19 illustrates the inclination a when the road boundary object is a side wall, and the absolute value of inclination a is 1.38, which is greater than or equal to 1.0, an example of a first inclination threshold, so the determination unit 213 may determine to set the recognition reliability of the road boundary object to wall level LV2. On the other hand, for example, Figure 20 illustrates the inclination a when the road boundary object is a curb, and the absolute value of inclination a is 0.88, which is greater than or equal to 0.8, an example of a second inclination threshold, and less than 1.0, an example of a first inclination threshold, so the determination unit 213 may determine to set the recognition reliability of the road boundary object to wall level LV1. The LV value of the wall level is set to be lower as the recognition reliability of the road boundary object decreases. However, this disclosure is not limited thereto, and for example, the LV value of the wall level LV can be set arbitrarily. The LV value of the wall level is set to be higher as the wall element becomes stronger, thus increasing the recognition reliability of the road boundary object. However, this disclosure is not limited to this, and for example, the LV value of the wall level can be set arbitrarily.

[0089] Furthermore, curb level LV0 to LV2 and wall level LV0 to LV2 are all examples of the recognition reliability of road boundary objects, and may be referred to as the "fourth recognition reliability of road boundary objects."

[0090] (2-3. First Operation Example of the Judgment Device) Referring to Figure 21, a first operation example of the judgment device 20 according to this embodiment will be described in accordance with the flowchart.

[0091] The first operational example is typically performed when the driver assistance system installed in vehicle 1 is operating. The following aims to explain the overall flow of the first operational example; for detailed processing of each step, please refer to the description of the functional configuration of the processing unit 21.

[0092] In step S11, the image acquisition unit 211 of the processing unit 21 acquires a stereo image from the stereo camera 10 that includes an object in front of the vehicle 1. As mentioned above, the object in front of the vehicle 1 may include road boundary objects in front of the vehicle 1. The process then proceeds to step S12.

[0093] In step S12, the mapping unit 215 of the processing unit 21 identifies a plurality of candidate points corresponding to road boundary objects in front of the vehicle 1 based on the disparity image generated by the image generation unit 214 based on the stereo image acquired in step S11. The identification of the plurality of candidate points can be performed, for example, by fitting the plurality of candidate points corresponding to road boundary objects mapped on the XZ plane with a quadratic curve, as shown in Figures 6 and 7. At this time, the mapping unit 215 may interpolate some of the candidate points among the plurality of candidate points corresponding to road boundary objects. After that, the process proceeds to step S13.

[0094] In step S13, the edge detection unit 212 of the processing unit 21 identifies the edge direction of the edges included in the stereo image based on the brightness change of the stereo image acquired in step S11. In this example, the edge direction is defined as edge direction 0, with the 12 o'clock direction in the forward field of view from the vehicle 1 being the edge direction 1, and edge directions 1, 2, 3, 4, 5, 6, and 7 identified at 45-degree intervals clockwise from edge direction 0. As a result, for example, the edge image shown in the lower part of Figure 3 is generated from the stereo image shown in the upper part of Figure 3. The process then proceeds to step S14.

[0095] In step S14, the edge detection unit 212 of the processing unit 21 sets up multiple regions along the multiple candidate points calculated in step S12. As a result, for example as shown in Figure 4, regions are set up centered on each of the Pi (where i is a natural number) along the curb between the lane and the curb that demarcate the driving area of ​​the vehicle 1. The process then proceeds to step S15.

[0096] In step S15, the determination unit 213 of the processing unit 21 determines the reliability of road boundary object recognition based on the edge directions of the edges included in the plurality of regions set in step S14, among the edge directions identified in step S13.

[0097] Specifically, the edge detection unit 212 calculates the ratio of the number of edges in the direction of travel of the vehicle 1, specifically the left-diagonal and right-diagonal edges, to the total number of edges in the multiple regions set in step S14. More specifically, the edge detection unit 212 calculates the left component ratio using equation (1) above, and the right component ratio using equation (2) above. Note that if interpolation is performed in step S12, the number of edges in the interpolated region is not added to the numerators of equation (1) and equation (2) above. The determination unit 213 then determines that if the left component ratio is greater than or equal to the first edge threshold, the first recognition confidence level of the road boundary object should be set to edge level LV2. The determination unit 213 also determines that if the left component ratio is greater than or equal to the second edge threshold and less than the first edge threshold, the first recognition confidence level of the road boundary object should be set to edge level LV1. Furthermore, the determination unit 213 determines that if the left component ratio is less than the second edge threshold, the first recognition confidence level of the road boundary object should be set to edge level LV0. On the other hand, if the right component ratio is greater than or equal to the first edge threshold, the determination unit 213 determines that the first recognition confidence level of the road boundary object should be set to edge level LV2. Furthermore, if the right component ratio is greater than or equal to the second edge threshold and less than the first edge threshold, the determination unit 213 determines that the first recognition confidence level of the road boundary object should be set to edge level LV1. Furthermore, if the right component ratio is less than the second edge threshold, the determination unit 213 determines that the first recognition confidence level of the road boundary object should be set to edge level LV0. Note that the "first recognition confidence level" may be expressed using a numerical value such as a score, and in this example, 2 points are assigned to "edge level LV2", 1 point is assigned to "edge level LV1", and 0 points are assigned to "edge level LV0".

[0098] According to the first example of operation, the reliability of road boundary object recognition is determined based on the edge direction of the edges included in multiple regions set along multiple candidate points corresponding to road boundary objects in the stereo image from the stereo camera 10. Therefore, by using this recognition reliability, the driver assistance system can determine whether the recognition of road boundary objects such as curbs or side walls has become unstable due to obstacles such as vegetation or snow. Thus, even if the recognition of road boundary objects using the stereo camera 10 becomes unstable, it is possible to suppress the misdetection of road boundary objects as objects other than their actual nature.

[0099] The first example of operation has been described above. However, among the processes in steps S11 to S15, the processes performed by components other than the basic configuration of the processing unit 21 may be omitted as appropriate, or replaced with known or arbitrary processes.

[0100] (2-4. Second Operation Example of the Judgment Device) Referring to Figure 22, a second operation example of the judgment device 20 according to this embodiment will be explained in accordance with the flowchart.

[0101] The second example of operation is an example of applying the determination result by the determination device 20 according to this embodiment, which is usually performed when the driver assistance system installed in the vehicle 1 is operating. However, this disclosure is not limited to this example and can be applied to various processes for the driver assistance system to recognize road boundary objects by stereo matching.

[0102] In step S21, the determination unit 213 of the processing unit 21 determines, in the same manner as in the first operation example, whether the LV value of the edge level corresponding to the first recognition confidence level is LV2, LV1, or LV0. In this operation example, 2 points are assigned to "edge level LV2", 1 point is assigned to "edge level LV1", and 0 points are assigned to "edge level LV0". The process then proceeds to step S22.

[0103] In step S22, the determination unit 213 of the processing unit 21 determines whether the LV value of the variation level corresponding to the second recognition confidence level is LV2, LV1, or LV0. Specifically, the determination unit 213 determines the variation level based on the variation from the calculated quadratic curve of a plurality of candidate points corresponding to road boundary objects mapped on the XZ plane by the mapping unit 215 as shown in Figures 7 and 8, in the same manner as described above. In this example, 2 points are assigned to "variation level LV2", 1 point is assigned to "variation level LV1", and 0 points are assigned to "variation level LV0". The process then proceeds to step S23.

[0104] In step S23, the determination unit 213 of the processing unit 21 determines whether the slope correlation level corresponding to the third recognition confidence level is slope correlation present or slope correlation absent. Specifically, the determination unit 213 determines the slope correlation level based on the slope a and correlation coefficient r of the linear approximation formula calculated by the slope correlation calculation unit 218, in the same manner as described above. In this example, 1 point is assigned to "slope correlation present" and 0 points are assigned to "slope correlation absent". The process then proceeds to step S24.

[0105] In step S24, the determination unit 213 of the processing unit 21 determines whether the LV value of the curb wall level corresponding to the fourth recognition confidence level is LV2, LV1, or LV0. Specifically, the determination unit 213 determines the curb level based on the degree of prominence of the frequency distribution generated by the frequency distribution generation unit 219, in the same manner as described above. On the other hand, the determination unit 213 determines the wall level based on the slope a calculated by the slope correlation calculation unit 218, in the same manner as described above. The determination unit 213 then determines that the LV value with the maximum among the LV values ​​of the curb level and the wall level should be set as the curb wall level.

[0106] More specifically, the determination unit 213 sets curb level LV2 as a candidate curb wall level if the first height class exists and the difference between the maximum and minimum Y coordinate positions of candidate points in the class to which the first height class belongs is greater than or equal to the height threshold. In this example, -2 points are assigned to "curb level LV2". If the first height class does not exist, the determination unit 213 sets curb level LV1 as a candidate curb wall level if the second height class exists and the difference between the maximum and minimum Y coordinate positions of candidate points in the class to which the second height class belongs is greater than or equal to the height threshold. In this example, -1 point is assigned to "curb level LV1". The reason for assigning a negative value is that curb levels LV1 and LV2 have a strong curb element and are elements that negate the HALT control described later.

[0107] On the other hand, if the absolute value of the slope a calculated by the slope correlation calculation unit 218 is greater than or equal to the first slope threshold, the determination unit 213 sets wall level LV2 as a candidate for curb wall level. In this example, "wall level LV2" is assigned -2 ​​points. If the absolute value of the slope a is greater than or equal to the second slope threshold, which is smaller than the first slope threshold, and less than the first slope threshold, the determination unit 213 sets wall level LV1 as a candidate for curb wall level. In this example, "wall level LV1" is assigned -1 point. If the absolute value of the slope a is less than the second slope threshold, the determination unit 213 sets wall level LV0 as a candidate for curb wall level. In this example, "wall level LV0" is assigned 0 points. The reason for assigning a negative value is that wall levels LV1 and LV2 have a strong wall element and are elements that negate the HALT control described later.

[0108] When the determination unit 213 determines the candidates for the curb wall level in this manner, it decides to set the LV value of the curb wall level to the highest LV value among the candidates for the curb wall level. Therefore, in this example, -2 points are assigned to "Curb wall level LV2", -1 point is assigned to "Curb wall level LV1", and 0 points are assigned to "Curb wall level LV0". The process then proceeds to step S25.

[0109] Furthermore, the processes in steps S21 to S24 do not necessarily have to be performed in the order shown in Figure 22. They may be swapped with each other, or some or all of the processes may be performed in parallel, as long as there is no logical contradiction.

[0110] In step S25, the determination unit 213 of the processing unit 21 determines whether or not to stop (HALT) the recognition of road boundary objects based on the determination results of steps S21 to S24. Specifically, the determination unit 213 determines to stop the recognition of road boundary objects if the sum of the first recognition confidence level, second recognition confidence level, third recognition confidence level, and fourth recognition confidence level remains above the HALT threshold for a predetermined period of time. On the other hand, if the sum of

[0111] If it is determined that the recognition of road boundary objects should be stopped (step S25: Y), the process proceeds to step S26. On the other hand, if it is not determined that the recognition of road boundary objects should be stopped (step S25: N), the process terminates. In other words, the recognition of road boundary objects by the driver assistance system continues.

[0112] In step S26, the processing unit 21 controls the driver assistance system to stop recognizing road boundary objects via the control device 7. The processing unit 21 may also notify the driver of vehicle 1 via the HMI 13 that the driver assistance system has stopped recognizing road boundary objects. The processing unit 21 may also control the vehicle 1 to automatically move to a safe location such as the shoulder of the road using known or arbitrary driver assistance technology via the control device 7.

[0113] According to the second example of operation, when determining whether or not to stop (HALT) recognition of road boundary objects using the stereo camera 10, the first recognition confidence level, second recognition confidence level, third recognition confidence level, and fourth recognition confidence level are comprehensively evaluated. Therefore, it is possible to suppress the occurrence of a HALT determination when the recognition of road boundary objects using the stereo camera 10 is in an unstable state.

[0114] Preferred embodiments of the present disclosure have been described in detail above with reference to the attached drawings, but the present disclosure is not limited to such examples. It is clear to any person with ordinary skill in the art to which the present disclosure belongs that various modifications or alterations can be conceived within the scope of the technical idea described in the claims, and these will naturally also be understood to fall within the technical scope of the present disclosure. For example, the functions etc. included in each component or each step etc. can be rearranged in a logically consistent manner, and multiple components or steps etc. can be combined into one or divided into two.

[0115] For example, the technology of this disclosure can be realized as a vehicle 1 equipped with the determination device 20 according to the above-described embodiment, and can also be realized as a determination method executed by a computer that can be configured in the same way as the determination device 20 according to the above-described embodiment. Furthermore, the technology of this disclosure can be realized as a determination program as a computer program that makes a computer that can be configured in the same way as the determination device 20 according to the above-described embodiment function, and can also be realized as a non-temporary tangible recording medium that records the determination program as a computer program.

[0116] 1: Vehicle, 10: Stereo camera, 20: Judgment device, 21: Processing unit, 211: Image acquisition unit, 212: Edge detection unit, 213: Judgment unit, 214: Image generation unit, 215: Correspondence unit, 216: Variation calculation unit, 217: Preprocessing unit, 218: Slope correlation calculation unit, 219: Frequency distribution generation unit, 22: Storage unit

Claims

1. A determination device for determining the reliability of recognition of a road boundary object in front of a vehicle, comprising one or more processors and one or more memories connected to the one or more processors, wherein the one or more processors acquire a stereo image including an object in front of the vehicle, identify the edge direction of an edge included in the stereo image based on the brightness change of the acquired stereo image, set a plurality of regions along a plurality of candidate points corresponding to the road boundary object in the acquired stereo image, and determine the reliability of recognition based on the edge direction of an edge included in the set plurality of regions from among the identified edge directions.

2. The determination device according to claim 1, wherein one or more processors calculate the total number of edges included in the plurality of regions, calculate the number of edges among the edges included in the plurality of regions whose edge direction is diagonally to the left and diagonally to the right in the direction of travel of the vehicle, and make a determination based on the ratio of the number of edges to the total number of edges when determining the recognition reliability.

3. The determination device according to claim 2, wherein, when interpolation is performed on some of the candidate points in identifying the plurality of candidate points, the one or more processors exclude edges included in the interpolated region from the determination of the recognition reliability when calculating the ratio.

4. The determination device according to claim 1, wherein one or more processors configure the plurality of regions such that the size of the plurality of regions decreases as the vehicle moves toward the rear in the direction of travel.

5. The determination device according to claim 1, wherein one or more processors generate a disparity image capable of determining the distance to the object based on the acquired stereo image; associate the plurality of candidate points corresponding to the road boundary object on a two-dimensional plane intersecting in the vehicle height direction based on the generated disparity image; calculate a quadratic curve by quadratic approximation of the associated plurality of candidate points; and, in determining the recognition reliability, make a determination based on the variation of the associated plurality of candidate points from the calculated quadratic curve.

6. The determination device according to claim 5, wherein one or more processors associate the plurality of candidate points with a further two-dimensional plane intersecting the vehicle's length direction, calculate the slope and correlation coefficient of a linear approximation formula by linearly approximating the plurality of candidate points associated with the further two-dimensional plane, and make a determination based on at least one of the calculated slope and correlation coefficient when determining the recognition reliability.

7. A determination method for determining the reliability of recognition of a road boundary object in front of a vehicle, the method comprising: a computer acquiring a stereo image including the object in front; identifying the edge direction of an edge included in the stereo image based on the brightness change of the acquired stereo image; setting a plurality of regions along a plurality of candidate points corresponding to the road boundary object in the acquired stereo image; and determining the reliability of recognition based on the edge direction of an edge included in the set plurality of regions from among the identified edge directions.

8. A non-temporary tangible recording medium on which a determination program for determining the reliability of recognition of a road boundary object in front of a vehicle is recorded, wherein the determination program is recorded which causes a computer to: acquire a stereo image including the object in front of the vehicle; identify the edge direction of an edge included in the stereo image based on the brightness change of the acquired stereo image; set up a plurality of regions along a plurality of candidate points corresponding to the road boundary object in the acquired stereo image; and determine the reliability of recognition based on the edge direction of an edge included in the set plurality of regions from among the identified edge directions.

Images