Determination device, determination method, and recording medium

Abstract

A determination device that determines a protruding shape on a snowy road in the area in front of a vehicle acquires a stereo image including an object in the area in front of the vehicle, generates, on the basis of the acquired stereo image, a parallax image with which the distance to the object can be identified, associates, on the basis of the generated parallax image, a plurality of candidate points corresponding to the object on a two-dimensional plane intersecting the vehicle length direction of the vehicle, and determines, on the basis of the plurality of candidate points, whether a protruding shape that projects in the vehicle height direction of the vehicle is present in a target range in the two-dimensional plane, said range extending to the left and right sides in the vehicle width direction from the vehicle width center of the vehicle.

Description

Determination Device, Determination Method, and Recording Medium

[0001] The present disclosure relates to a determination device, a determination method, and a recording medium. In particular, the present disclosure relates to a determination device, a determination method, and a recording medium for determining a convex shape on a snowy road ahead of a vehicle.

[0002] Conventionally, a technique for providing driving support for a vehicle on a snowy road where tracks are formed has been known.

[0003] For example, Patent Document 1 discloses an image acquisition unit that acquires a road surface image obtained by photographing the road surface around a vehicle, a detection unit that detects a lane boundary line by image processing from the road surface image acquired by the image acquisition unit, a calculation unit that calculates a parameter indicating the degree of whiteness of a region corresponding to below the center of the vehicle compared to a region corresponding to below the tire of the vehicle in the road surface image acquired by the image acquisition unit, and at least based on the parameter calculated by the calculation unit, a determination unit that determines whether to execute any one of the detection by the detection unit, the output of the detected lane boundary line, and a predetermined control using the detected lane boundary line. A lane boundary line detection device characterized by comprising the above is disclosed.

[0004] Japanese Patent Application Laid-Open No. 2013-250874

[0005] In the conventional technology disclosed in Patent Document 1, a portion of the snowy road where there is no snow appears black like asphalt, while a portion where there is snow appears white due to the snow. Therefore, a parameter indicating the degree of whiteness is introduced as an index for the formation of snow tracks. However, when the road surface condition deteriorates further due to snow accumulation, the determination based on the parameter indicating the degree of whiteness is not sufficient, and there was room for improvement in appropriately providing driving support for a vehicle on a snowy road where tracks are formed.

[0006] In view of such circumstances, an object of the present disclosure is to provide a technique for appropriately providing driving support for a vehicle on a snowy road where tracks are formed.

[0007] A determination device according to one embodiment of the present disclosure is a determination device for determining a convex shape on a snowy road in front of a vehicle, comprising one or more processors and one or more memories connected to the one or more processors, wherein the one or more processors acquire a stereo image including an object in front, generate a disparity image capable of determining the distance to the object based on the stereo image, associate a plurality of candidate points corresponding to the object on a two-dimensional plane intersecting the vehicle's length direction based on the disparity image, and determine whether or not a convex shape protruding in the vehicle's height direction exists in a target range on both the left and right sides of the two-dimensional plane along the vehicle width direction from the center of the vehicle's width, based on the plurality of candidate points.

[0008] A determination method according to one embodiment of the present disclosure is a determination method for determining a convex shape on a snowy road in front of a vehicle, comprising: a computer acquiring a stereo image including an object in front of the vehicle; generating a parallax image capable of specifying the distance to the object based on the stereo image; associating a plurality of candidate points corresponding to the object with a two-dimensional plane intersecting the vehicle's length direction based on the parallax image; and determining, based on the plurality of candidate points, whether or not a convex shape protruding in the vehicle's height direction exists in a target range on both the left and right sides of the two-dimensional plane along the vehicle width direction from the center of the vehicle's width.

[0009] A recording medium according to one embodiment of the present disclosure is a non-temporary tangible recording medium on which a determination program for determining a convex shape on a snowy road in front of a vehicle is recorded, the recording medium having a computer perform the following actions: to cause a computer to acquire a stereo image including an object in front of the vehicle; to generate a parallax image capable of determining the distance to the object based on the stereo image; to associate a plurality of candidate points corresponding to the object with a two-dimensional plane intersecting the vehicle's length direction based on the parallax image; and to determine whether or not a convex shape protruding in the vehicle's height direction exists in a target range on both the left and right sides of the two-dimensional plane along the vehicle's width from the center of the vehicle's width.

[0010] According to one embodiment of the present disclosure, it is possible to appropriately assist the driving of a vehicle on a snowy road where ruts have formed.

[0011] This is a schematic diagram showing an example of the configuration of a vehicle equipped with a determination device according to one embodiment of the present disclosure. This is a block diagram showing an example of the configuration of a determination device according to one embodiment of the present disclosure. This is a diagram illustrating the mapping of multiple candidate points to the XY plane. This is a diagram illustrating the process of narrowing down multiple candidate points. This is a diagram illustrating the process of removing isolated points from multiple candidate points. This is a diagram illustrating the process of removing isolated points from multiple candidate points. This is a diagram illustrating multiple candidate points from which isolated points have been removed. This is a diagram illustrating the frequency distribution. This is a diagram illustrating the maximum value distribution. This is a diagram illustrating an example of HALT determination. This is a diagram illustrating an example of HALT determination. This is a flowchart illustrating an example of the operation of a determination device according to one embodiment of the present disclosure.

[0012] Preferred embodiments of this disclosure will be described in detail below with reference to the attached drawings. In this specification and the drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant descriptions will be omitted.

[0013] (1. Vehicle) Referring to Figure 1, Vehicle 1 is configured as a four-wheeled automobile that transmits the drive torque output from the drive source 2 to the wheels. Vehicle 1 may be an automobile equipped with an internal combustion engine such as a gasoline engine or a diesel engine as the drive source 2, or it may be an electric vehicle equipped with a drive motor as the drive source 2. Examples of electric vehicles include BEV (Battery Electric Vehicle), HEV (Hybrid Electric Vehicle), PHEV (Plug-in Hybrid Electric Vehicle), or FCEV (Fuel Cell Electric Vehicle).

[0014] The drive source 2 outputs drive torque which is transmitted to the front wheel drive shaft 4F via a transmission (not shown) and a differential mechanism 3. The drive of the drive source 2 and the transmission is controlled by a control device 7, which will be described later. The combination of drive wheels and the driving method are not particularly limited, and the vehicle 1 may be a front-wheel drive vehicle, a rear-wheel drive vehicle, or a four-wheel drive vehicle. Furthermore, if the vehicle 1 is configured as an electric vehicle, it may be an electric vehicle equipped with a drive motor corresponding to each wheel.

[0015] Vehicle 1 is equipped with, in addition to the aforementioned drive force source 2, at least a steering device 5, a brake device 6, and a control device 7 as equipment used for driving control.

[0016] The steering device 5 is mounted on the front wheel drive shaft 4F. The steering device 5 may include an electric motor (not shown) and a gear mechanism (not shown), and adjusts the steering angle of the front wheels by being controlled by the control device 7.

[0017] The brake device 6 applies braking force to the wheels under the control of the control device 7. If the vehicle 1 is configured as an electric vehicle, the brake device 6 may be used in conjunction with regenerative braking provided by the drive motor, which serves as the driving force source 2.

[0018] The control device 7 includes at least one or more ECUs (Electronic Control Units) that control the driving of the power source 2, the steering device 5, and the brake device 6, respectively.

[0019] During manual operation, the control device 7 calculates the drive torque to be output to the drive source 2 based on the amount of operation of the accelerator pedal (not shown) by the driver. Also, during manual operation, the control device 7 calculates the braking force to be output to the brake device 6 based on the amount of operation of the brake pedal (not shown) by the driver. Also, during manual operation, the control device 7 controls the steering device 5 based on the steering angle of the steering wheel 8 by the driver.

[0020] During autonomous driving, the control device 7 controls the drive force source 2, steering device 5, and brake device 6, respectively, based on the target vehicle speed and target steering angle of the vehicle 1, which are appropriately set using known or arbitrary driver assistance technologies. However, the control device 7 is connected to a determination device 20, which will be described later, via a dedicated line or a communication means such as CAN (Controller Area Network) or LIN (Local Internet), and may perform control such as pulling the vehicle 1 to the side of the road according to the determination result of the determination device 20. Advanced Driver-Assistance Systems (ADAS) are examples of driver assistance technologies, but this disclosure is not limited thereto.

[0021] Vehicle 1 further includes a stereo camera 10, a vehicle state sensor 11, and a position detection sensor 12.

[0022] The stereo camera 10 includes a main camera 10a and a sub-camera 10b. The main camera 10a and the sub-camera 10b each include an image sensor such as a CCD (Charged Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor). The main camera 10a and the sub-camera 10b are each connected to the determination device 20 by wired or wireless communication means, and to the control device 7 as needed. The main camera 10a and the sub-camera 10b each capture the area in front of the vehicle 1 to generate a stereo image, transmit the generated stereo image to the determination device 20, and transmit it to the control device 7 as needed. Note that in Figure 1, the main camera 10a and the sub-camera 10b may be configured with their left and right sides reversed.

[0023] Specifically, the main camera 10a generates a reference image at a predetermined frame rate, and the sub-camera 10b generates a comparison image at a predetermined frame rate. As will be described in detail later, the determination device 20 acquires the reference image generated by the main camera 10a and the comparison image generated by the sub-camera 10b as stereo images. Furthermore, the determination device 20 generates a disparity image that can determine the distance to objects included in the stereo image by applying image processing such as stereo matching to the acquired stereo image.

[0024] In addition to the stereo camera 10, vehicle 1 may also be equipped with a camera (not shown) for imaging the area behind vehicle 1. Furthermore, vehicle 1 may also be equipped with one or more distance measuring sensors (not shown) from among radar sensors such as LiDAR (Light Detection And Ranging) or millimeter-wave radar and ultrasonic sensors.

[0025] The vehicle state sensor 11 is a known or arbitrary sensor for detecting the state of the vehicle 1. For example, the vehicle state sensor 11 may include a vehicle speed sensor for detecting the speed of the vehicle 1, and may include a steering angle sensor for detecting the steering angle of the steering wheel 8. The vehicle state sensor 11 transmits the detection result to the control device 7 and, if necessary, to the determination device 20. The detection result from the vehicle state sensor 11 is used as appropriate in the driving assistance of the vehicle 1 by the control device 7.

[0026] The position detection sensor 12 may include a GNSS (Global Navigation Satellite System) sensor that receives satellite signals from positioning satellites such as GPS (Global Positioning System) satellites. The position detection sensor 12 transmits information indicating the current position of the vehicle 1, which is included in the received satellite signals, to the control device 7, and to the determination device 20 as necessary.

[0027] In addition, vehicle 1 may further include a known or arbitrary HMI (Human Machine Interface) 13 that presents various information to the driver of vehicle 1 through image or text display or audio output, etc.

[0028] (2. Determination Device) The determination device 20 according to this embodiment will be described in detail with reference to Figure 2.

[0029] (2-1. Configuration Example) The determination device 20 functions as a device that determines the shape of a convex shape on the snowy road in front of the vehicle 1 by having one or more CPUs (Central Processing Units) or other processors execute a computer program. The computer program is a computer program that causes the processor to execute the operations that the determination device 20 should perform, which will be described later. The computer program executed by the processor may be recorded on a recording medium that functions as a storage unit (memory) 22, which will be described later, or it may be recorded on a recording medium built into the determination device 20 or on any recording medium that can be attached externally to the determination device 20.

[0030] The recording medium for storing computer programs may include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs, DVDs, and Blu-ray®; magneto-optical media such as floppy disks; memory elements such as RAM (Random Access Memory) and ROM (Read Only Memory); flash memory such as USB (Universal Serial Bus) memory and SSD (Solid State Drive); and other media capable of storing programs.

[0031] As described above, the determination device 20 is connected to at least the control device 7 and the stereo camera 10. It is also possible that some or all of the components of the determination device 20 are provided on the control device 7 side, or that they are provided on the server (not shown) side capable of communicating with the vehicle 1.

[0032] The determination device 20 comprises at least a processing unit 21 and a storage unit 22. If some or all of the components of the determination device 20 are provided on the server side, the determination device 20 further comprises a communication unit (not shown) which includes a communication interface for communicating with the vehicle 1 via a known or arbitrary network (not shown).

[0033] (Processing Unit) The processing unit 21 comprises one or more processors such as CPUs and various peripheral components. Part or all of the processing unit 21 may consist of updatable components such as firmware, or it may be a program module that is executed by instructions from the CPU, etc.

[0034] (Storage Unit) The storage unit 22 is composed of one or more storage elements such as RAM or ROM that are connected to the processing unit 21 in a communicative manner. However, the type and number of storage units 22 are not particularly limited. The storage unit 22 stores information such as computer programs executed by the processing unit 21, various parameters used in arithmetic processing, detection results, and calculation results.

[0035] (2-2. Functional Configuration of the Processing Unit) The functional configuration of the processing unit 21 of the determination device 20 will be described below. The processing unit 21 comprises at least an image acquisition unit 211, an image generation unit 212, a mapping unit 213, and a shape determination unit 214. Each of the image acquisition unit 211, image generation unit 212, mapping unit 213, and shape determination unit 214 is a function realized by the execution of a computer program by one or more processors such as a CPU. However, some or all of the image acquisition unit 211, image generation unit 212, mapping unit 213, and shape determination unit 214 may be configured using analog circuits.

[0036] (Image Acquisition Unit) The image acquisition unit 211 acquires a stereo image including an object in front of the vehicle 1. Specifically, the image acquisition unit 211 may acquire a reference image including an object in front of the vehicle 1, captured by the main camera 10a included in the stereo camera 10, and a comparison image including an object in front of the vehicle 1, captured by the sub-camera 10b included in the stereo camera 10, as a stereo image. It is preferable that the image acquisition unit 211 acquires the stereo image at a calculation cycle synchronized with the frame rate of the stereo camera 10, and the acquired stereo image may be stored in the storage unit 22 linked to the acquisition time.

[0037] (Image Generation Unit) The image generation unit 212 generates a parallax image capable of determining the distance to an object in front of the vehicle 1 based on the stereo image acquired by the image acquisition unit 211. Specifically, the image generation unit 212 applies known or arbitrary stereo matching processing to the reference image and comparison image as stereo images. More specifically, the image generation unit 212 searches the comparison image for blocks corresponding to blocks arbitrarily extracted from the reference image, and calculates the parallax of the blocks in the reference image from the positional relationship between the blocks in the reference image and the blocks in the comparison image. It is preferable that the image generation unit 212 generates a parallax image by performing this series of processing on all blocks included in the reference image. It is preferable that the image generation unit 212 generates a parallax image each time the image acquisition unit 211 acquires a stereo image, and the generated parallax image may be stored in the storage unit 22 in association with the acquisition time of the stereo image. The value or color of each block in the parallax image indicates the parallax and corresponds to the distance to the block in three-dimensional real space corresponding to each block. Furthermore, while each block is exemplified as a 4x4 pixel configuration, this disclosure is not limited to this and can be set arbitrarily.

[0038] (Mapping section: XY plane) The mapping section 213 maps a plurality of candidate points corresponding to objects in front of the vehicle 1 onto a two-dimensional plane that intersects the vehicle length direction of the vehicle 1, based on the disparity image generated by the image generation section 212. The objects in front of the vehicle 1 may include convex shapes, such as those caused by snow ruts on a snowy road in front of the vehicle 1.

[0039] Specifically, the coordinates (i, j) and parallax of each block in the parallax image are mapped one-to-one with the coordinates (x, y, z) of each block in a three-dimensional real space where the vehicle width direction of the vehicle 1 is the X-axis, the vehicle height direction is the Y-axis, and the vehicle length direction is the Z-axis, by coordinate transformation based on the principle of triangulation. The mapping unit 213 then applies the principle of triangulation to the parallax image generated by the image generation unit 212. As a result, the mapping unit 213 maps multiple candidate points corresponding to objects in front of the vehicle 1 onto the XY plane, for example, as shown in Figure 3. Preferably, each time the image generation unit 212 generates a parallax image, the mapping unit 213 uses the generated parallax image to map it onto the XY plane at predetermined intervals (e.g., 2 m) along the Z-axis, and the result of the mapping onto the XY plane may be stored in the storage unit 22 in association with the acquisition time of the stereo image.

[0040] In the example shown in Figure 3, multiple candidate points corresponding to an object at a point 8 m (Z = 8 m) in front of vehicle 1 are mapped onto the XY plane. In Figure 3, the point group consisting of multiple candidate points existing along the X-axis from near X = -1.5 m to near X = 3.0 m corresponds to the road surface on which vehicle 1 is traveling. Furthermore, the point groups consisting of multiple candidate points that are inclined to rise from the road surface from near X = -1.5 m to near X = -2.0 m, and from near X = 3.0 m to near X = 3.5 m, correspond to snow walls, respectively. In addition, the convex shape existing from near X = -0.5 m to near X = 0.0 m corresponds to snow tracks.

[0041] (Mapping section: XZ plane) The mapping section 213 may map a plurality of candidate points corresponding to curb candidates in front of the vehicle 1 onto a further two-dimensional plane that intersects in the vehicle height direction of the vehicle 1, based on the disparity image generated by the image generation section 212. The mapping section 213 may also calculate curb candidate approximation lines CL and CR, which are formed by fitting the plurality of candidate points mapped onto the further two-dimensional plane. The mapping section 213 may also map the calculated curb candidate approximation lines CL and CR onto a two-dimensional plane that intersects in the vehicle length direction of the vehicle 1.

[0042] Specifically, the associating unit 213 may associate a plurality of candidate points corresponding to curb candidates in front of the vehicle 1 on the XZ plane by applying the principle of triangulation to the parallax image generated by the image generation unit 212. Further, the associating unit 213 may calculate curb candidate approximation lines CL and CR formed by, for example, a quadratic curve by applying a known or arbitrary fitting method such as quadratic approximation by the least squares method or the like to the plurality of candidate points associated on the XZ plane. Further, the associating unit 213 may associate the calculated curb candidate approximation lines CL and CR on the XY plane at intervals of 2 m over a range from, for example, Z = 8 to Z = 12 m. Note that the curb candidates are not necessarily limited to curbs and may include, for example, snow tracks as will be described later.

[0043] (Shape determination unit) The shape determination unit 214 determines whether or not there is a convex shape protruding in the vehicle height direction of the vehicle 1 based on a plurality of candidate points in a target range T on both the left and right sides along the vehicle width direction from the vehicle width center W of the vehicle 1 in the two-dimensional plane in which the associating unit 213 associates the plurality of candidate points. Specifically, the shape determination unit 214 determines whether or not there is a convex shape protruding in the Y direction based on a plurality of candidate points included in a range corresponding to the vehicle width W including the vehicle width center W of the vehicle 1 in the X direction and in a range excluding the vicinity of the road surface in the Y direction in the XY plane corresponding to the two-dimensional plane. Note that the convex shape is caused by, for example, a snow track and does not necessarily have to protrude parallel to the Y direction in a mathematically strict sense as long as it protrudes from the road surface on which the vehicle 1 is traveling. ０ to ０

[0044] As described above, the basic configuration of the processing unit 21 of the determination device 20 has been described. However, the processing unit 21 may further include a preprocessing unit 215, a distribution generation unit 216, and a HALT determination unit 217. Each of the preprocessing unit 215, the distribution generation unit 216, and the HALT determination unit 217 is a function realized by executing a computer program by one or a plurality of processors such as a CPU. However, a part or all of the preprocessing unit 215, the distribution generation unit 216, and the HALT determination unit 217 may be configured using an analog circuit.

[0045] ​(Preprocessing Unit) From the perspective of reducing processing time and processing load, the preprocessing unit 215 may perform the following narrowing as preprocessing on the plurality of candidate points associated by the association unit 213 on the two-dimensional plane. That is, the preprocessing unit 215 may narrow down the candidate points included in the target range T from among the plurality of candidate points corresponding to the objects in front of the vehicle 1 associated by the association unit 213 on the two-dimensional plane.

[0046] Specifically, for example, as shown in the shaded area of FIG. 4, on the XY plane corresponding to the two-dimensional plane, in the X direction, the range corresponding to the vehicle width W including the vehicle width center W of the vehicle 1 ０ and, in the Y direction, the range excluding the vicinity of the road surface may be specified as the target range T. Further, the preprocessing unit 215 may narrow down the candidate points included in the target range T from among the plurality of candidate points on the XY plane.

[0047] The vicinity of the road surface is not particularly limited, but the Y coordinate positions up to 30 mm in the Y direction from the road surface that can be specified by applying a known or arbitrary fitting technique for specifying the road surface shape to the plurality of candidate points associated on the XY plane are exemplified. The vehicle width can be appropriately set according to the vehicle type of the vehicle 1. For example, in FIG. 4, a plurality of candidate points at a point 8 m (Z = 8 m) in front of the vehicle 1 are associated on the XY plane, and the range from the vicinity of X = -0.9 m around the vicinity of X = 0.0 m to the vicinity of X = 0.9 m and the range from the vicinity of Y = 30 mm to the vicinity of Y = 230 mm are shown as the target range T by shading.

[0048] As a result, the candidate points that exist below the vicinity of the road surface and outside the vehicle width W of the vehicle 1 will be deleted. By deleting the candidate points that exist outside the vehicle width W of the vehicle 1, it is possible to avoid the HALT determination described later from being performed even if the vehicle 1 approaches, for example, a curb.

[0049] The preprocessing unit 215 may further perform preprocessing to remove isolated points from candidate points included in the target range T, in order to improve the accuracy of determining convex shapes without increasing the processing load. Specifically, the preprocessing unit 215 may further divide the target range T into multiple cells, identify isolated points from among the candidate points included in the target range T based on the number of candidate points included in each cell, and perform preprocessing to remove the identified isolated points.

[0050] More specifically, the preprocessing unit 215 may further divide the target range T into multiple cells by setting a rectangular cell with a side length of 50 mm along the X-axis and a side length of 10 mm along the Y-axis, as shown in Figure 5. The preprocessing unit 215 may also calculate the sum of candidate points contained in a total of nine cells: a target cell C arbitrarily extracted from such multiple cells, and eight surrounding cells C' that surround the target cell C. Furthermore, if the calculated sum of candidate points is less than the deletion threshold, the preprocessing unit 215 may identify the candidate points contained in the target cell C as isolated points and delete the identified isolated points.

[0051] In the example shown in Figure 5, the deletion threshold is 3, and the sum of the candidate points in the nine cells consisting of target cell C1 and surrounding cells C1' is 3, so the candidate points in target cell C1 are not deleted. On the other hand, the sum of the candidate points in the nine cells consisting of target cell C2 and surrounding cells C2' is 2, so the candidate points in target cell C2 are deleted. Also, the sum of the candidate points in the nine cells consisting of target cell C3 and surrounding cells C3' is 2, so the candidate points in target cell C3 are deleted. It is preferable that the preprocessing unit 215 performs this preprocessing on all cells included in the target range T. As a result, for example, candidate points included in the solid thick frame within the target range T shown by the shaded area in Figure 6 are identified as isolated points and deleted, for example, as shown in Figure 7.

[0052] (Distribution generation unit) From the viewpoint of improving the accuracy of determining convex shapes, the distribution generation unit 216 may perform the following two processes when the shape determination unit 214 determines a convex shape.

[0053] In other words, the distribution generation unit 216 may generate a frequency distribution in the target range T in which predetermined intervals along the vehicle width direction of the vehicle 1 are used as classes, and the number of candidate points included in the predetermined interval among a plurality of candidate points is the frequency. Specifically, as shown in Figure 8, for example, the distribution generation unit 216 may generate a frequency distribution in which each interval when the X-axis is divided into 50 mm intervals in the target range T identified by the preprocessing unit 215 is used as a class, and the number of candidate points in each class is the frequency. Note that the predetermined interval is not necessarily limited to 50 mm and can be set as appropriate.

[0054] The distribution generation unit 216 may identify the height of the candidate point where the height along the vehicle height direction of the vehicle 1 is maximum for each class of the frequency distribution as a partial maximum value, and generate a maximum value distribution composed of the partial maximum values ​​for each class. Specifically, the distribution generation unit 216 may identify the Y coordinate position of the candidate point where the Y coordinate position is maximum for each class of the frequency distribution corresponding to each interval when the X axis is divided into 50 mm intervals as a partial maximum value. Alternatively, the distribution generation unit 216 may generate a maximum value distribution composed of the partial maximum values ​​for each class identified in this way, as shown in Figure 9, for example.

[0055] The frequency distribution and maximum value distribution generated in this way can be used by the shape determination unit 214 to determine whether a convex shape exists. That is, the shape determination unit 214 may determine that a convex shape exists if the maximum value in the maximum value distribution shown in Figure 9 is greater than or equal to the height threshold, and the frequency in the frequency distribution shown in Figure 8 of the class to which the maximum value belongs is greater than or equal to the frequency threshold. More preferably, the shape determination unit 214 designates the class in which the maximum value in the maximum value distribution shown in Figure 9 is greater than or equal to the height threshold, and the frequency in the frequency distribution shown in Figure 8 is greater than or equal to the frequency threshold as the representative class (the thick hatched areas in Figures 8 and 9). The shape determination unit 214 also designates the class adjacent to the representative class as the adjacent class (the thin hatched areas in Figures 8 and 9). In this case, the shape determination unit 214 may determine whether a convex shape exists based on the partial maximum value of the representative class in the maximum value distribution (i.e., the maximum value in the maximum value distribution) and the partial maximum value of the adjacent class in the maximum value distribution. More specifically, the shape determination unit 214 may determine that a convex shape exists if the ratio of the partial maximum value of an adjacent class to the partial maximum value of a representative class is equal to or greater than the ratio threshold. The height threshold is, for example, 80 mm, the frequency threshold is, for example, 5, and the ratio threshold is, for example, 1 / 2, but this disclosure is not limited to these and can be set as appropriate.

[0056] (HALT determination unit) If the shape determination unit 214 determines that a convex shape exists, the HALT determination unit 217 may determine to stop recognizing objects in front of the vehicle 1 (HALT). In this case, the processing unit 21 may notify the driver of the vehicle 1 via the vehicle 1's HMI 13 that object recognition by the driver assistance system has stopped. The HALT determination unit 217 may further perform the following three processes.

[0057] In other words, the HALT determination unit 217 may decide to stop recognizing an object if the shape determination unit 214 determines that the curb candidate point approximation lines CL and CR, which are mapped onto a two-dimensional plane by the correspondence unit 213, are not included in the target range T, and a convex shape exists. Specifically, for example as shown in Figure 10, if (i) a convex shape exists in the forward region T' from 8m to 12m in front of the vehicle 1, and (ii) both curb candidate approximation lines CL and CR exist outside the forward region T', the HALT determination unit 217 may decide to stop recognizing an object. In other words, the situation shown in Figure 10 is a situation in which there is a high possibility that the vehicle 1 is driving on snow, and the snow wall may be mistakenly detected as a side wall, and the road surface friction may be very low. In such a case, by using the determination result of the HALT determination unit 217, the steering angle control by the control device 7 of the vehicle 1 can be stopped, thereby ensuring safety. Furthermore, the cross-sections that intersect the forward region T' at Z=8m, Z=10m, and Z=12m each correspond to the target range T described above. In other words, this determination may be performed at the three locations of Z=8m, Z=10m, and Z=12m. The same applies hereafter.

[0058] Furthermore, the HALT determination unit 217 may determine to stop object recognition if the curb candidate approximation lines CL and CR, which have been mapped onto a two-dimensional plane by the mapping unit 213, are included in the target range T, and the current position of vehicle 1 lies on the curb candidate approximation lines CL and CR. Specifically, for example as shown in Figure 11, if (i) for example, curb candidate approximation line CL is located inside the forward region T' from 8m to 12m in front of vehicle 1, and (ii) for example, the wheels of vehicle 1 are running over curb candidate approximation line CL, then the HALT determination unit 217 may determine that vehicle 1 is already running over snow tracks, not a curb, and stop object recognition. This prevents the driver assistance system from mistakenly detecting a snow wall as a curb and controlling the steering angle of vehicle 1.

[0059] Furthermore, the HALT determination unit 217 may determine that it will not stop recognizing the object if the curb candidate approximation lines CL and CR, which have been mapped onto a two-dimensional plane by the mapping unit 213, are included in the target range T, and the current position of the vehicle 1 is not on the curb candidate approximation lines CL and CR. Specifically, for example as shown in Figure 12, if (i) for example the curb candidate approximation line CR is located inside the forward region T' from 8m to 12m in front of the vehicle 1, and (ii) the wheels of the vehicle 1 are not touching either of the curb candidate approximation lines CL or CR, the HALT determination unit 217 does not need to determine that it will stop recognizing the object. This prevents the driver assistance system from mistakenly detecting the curb as a snow rut and failing to control the steering angle of the vehicle 1 when the vehicle 1 is traveling on a curve.

[0060] (2-3. Example of Operation of the Judgment Device) Referring to Figure 13, an example of the operation of the judgment device 20 according to this embodiment will be explained in accordance with the flowchart.

[0061] This example of operation is typically performed when the driver assistance system installed in vehicle 1 is in operation. The following aims to explain the overall flow of this example of operation; for detailed processing of each step, please refer to the description of the functional configuration of processing unit 21.

[0062] In step S11, the image acquisition unit 211 of the processing unit 21 acquires a stereo image from the stereo camera 10 that includes an object in front of the vehicle 1. The object in front of the vehicle 1 may include a convex shape caused by snow ruts on the snowy road in front of the vehicle 1. The process then proceeds to step S12.

[0063] In step S12, the image generation unit 212 of the processing unit 21 generates a disparity image that can determine the distance to an object in front of the vehicle 1, based on the stereo image acquired in step S11. The process then proceeds to step S13.

[0064] In step S13, the correspondence unit 213 of the processing unit 21 associates multiple candidate points corresponding to objects in front of the vehicle 1 onto the XY plane, for example, as shown in Figure 3, based on the disparity image generated in step S12. The XY plane is an example of a two-dimensional plane that intersects the vehicle length direction of the vehicle 1. The process then proceeds to step S14.

[0065] In step S14, the preprocessing unit 215 of the processing unit 21 narrows down the candidate points included in the target range T from among the multiple candidate points mapped to the XY plane in step S13, for example as shown in Figure 4. The target range T is defined as the center W of the vehicle width of the vehicle 1 on the XY plane, as set by the preprocessing unit 215 as described above. ０ This range extends along the width of the vehicle on both the left and right sides, and may include convex shapes caused by snow ruts. The process then proceeds to step S15.

[0066] In step S15, the preprocessing unit 215 of the processing unit 21 identifies isolated points from the candidate points narrowed down in step S14, for example as shown in Figure 6. Then, the preprocessing unit 215 deletes the identified isolated points, for example as shown in Figure 7. After that, the process proceeds to step S16.

[0067] In step S16, the shape determination unit 214 of the processing unit 21 determines whether or not a convex shape exists that protrudes in the Y direction, which corresponds to the vehicle height direction of the vehicle 1, based on a plurality of candidate points from which isolated points were removed in step S15. In determining whether or not a convex shape exists, the frequency distribution and maximum value distribution generated by the distribution generation unit 216 as described above may be used. If it is determined that a convex shape exists (step S16: Y), the process proceeds to step S17. On the other hand, if it is not determined that a convex shape exists (step S16: N), the process ends. That is, the recognition of objects in front of the vehicle 1 by the driver assistance system continues.

[0068] In step S17, the HALT determination unit 217 of the processing unit 21 determines to stop recognizing (HALT) an object in front of the vehicle 1. The HALT determination unit 217 may also determine to stop recognizing an object in front of the vehicle 1 if the state in which the shape determination unit 214 determines that a convex shape exists continues for a predetermined period of time or longer. After that, the process ends.

[0069] Furthermore, among the processes in steps S11 to S17, processes performed by components other than the basic configuration of the processing unit 21 may be omitted as appropriate.

[0070] According to this example, the center W of the vehicle width of vehicle 1 is in the XY plane, which corresponds to a two-dimensional plane. ０ Based on multiple candidate points included in the target range T on both the left and right sides along the vehicle width direction, it is possible to determine whether or not there is a convex shape protruding in the vehicle height direction of the vehicle 1. Therefore, even when road conditions are poor due to snow, the driving assistance system can use this determination result to avoid misdetecting, for example, a tall object appearing on the road surface inside the side wall as a curb and performing steering control of the vehicle 1. Thus, it is possible to appropriately assist the driving of the vehicle 1 on snowy roads where ruts have formed.

[0071] However, this example of operation is just one example of how the determination result by the determination device 20 according to this embodiment can be applied, and this disclosure is not limited thereto. It can be applied to various processes for a driver assistance system to recognize snow tracks by stereo matching.

[0072] While preferred embodiments of this disclosure have been described in detail above with reference to the attached drawings, this disclosure is not limited to such examples. It is clear to any person with ordinary skill in the art to which this disclosure pertains that various modifications or alterations can be conceived within the scope of the technical idea set forth in the claims, and these too are understood to fall within the technical scope of this disclosure. For example, the functions, etc., included in each component or step can be rearranged in a logically consistent manner, and multiple components or steps can be combined into one or divided into separate components.

[0073] For example, the technology of this disclosure can be realized as a vehicle 1 equipped with the determination device 20 according to the above-described embodiment, and can also be realized as a determination method executed by a computer that can be configured in the same way as the determination device 20 according to the above-described embodiment. Furthermore, the technology of this disclosure can be realized as a determination program as a computer program that makes a computer that can be configured in the same way as the determination device 20 according to the above-described embodiment function, and can also be realized as a non-temporary tangible recording medium that records the determination program as a computer program.

[0074] 1: Vehicle, 10: Stereo camera, 20: Judgment device, 21: Processing unit, 211: Image acquisition unit, 212: Image generation unit, 213: Correspondence unit, 214: Shape determination unit, 215: Preprocessing unit, 216: Distribution generation unit, 217: HALT determination unit, 22: Storage unit

Claims

1. A determination device for determining a convex shape on a snowy road in front of a vehicle, comprising one or more processors and one or more memories connected to the one or more processors, wherein the one or more processors acquire a stereo image including an object in front of the vehicle, generate a disparity image capable of determining the distance to the object based on the stereo image, associate a plurality of candidate points corresponding to the object on a two-dimensional plane intersecting the vehicle's length direction based on the disparity image, and determine whether or not the convex shape protruding in the vehicle's height direction exists in a target range on both the left and right sides of the two-dimensional plane along the vehicle's width from the center of the vehicle's width, based on the plurality of candidate points.

2. The determination device according to claim 1, wherein one or more processors generate a frequency distribution in the target range, where predetermined intervals along the vehicle width direction are defined as classes and the number of candidate points included in the predetermined intervals among the plurality of candidate points is defined as the frequency; for each class of the frequency distribution, the height of the candidate point with the maximum height along the vehicle height direction is identified as a partial maximum; a maximum value distribution is generated from the partial maximum values ​​for each class; and if the maximum value in the maximum value distribution is greater than or equal to a height threshold, and the frequency in the frequency distribution of the class to which the maximum value belongs is greater than or equal to a frequency threshold, the device determines that the convex shape exists.

3. The determination device according to claim 2, wherein one or more processors determine whether the convex shape exists based on the partial maximum value of the representative class and the partial maximum value of the adjacent class in the maximum value distribution, with the class in which the maximum value is equal to or greater than the height threshold and the frequency is equal to or greater than the frequency threshold being designated as a representative class, and the classes adjacent to the representative class being designated as adjacent classes.

4. The determination device according to any one of claims 1 to 3, wherein one or more processors determine that the convex shape exists and decide to stop recognizing the object.

5. The determination device according to claim 4, wherein one or more processors associate a plurality of candidate points corresponding to the curb candidate in front of the vehicle with a further two-dimensional plane intersecting in the vehicle height direction based on the disparity image, calculate a curb candidate approximation line formed by fitting the plurality of candidate points associated with the further two-dimensional plane, and when the curb candidate approximation line is associated with the two-dimensional plane intersecting in the vehicle length direction, the device determines that the curb candidate approximation line is not included in the target range and that the convex shape exists, and determines to stop the recognition of the object.

6. The determination device according to claim 4, wherein one or more processors associate a plurality of candidate points corresponding to the curb candidate in front of the vehicle with a further two-dimensional plane intersecting in the vehicle height direction based on the disparity image, calculate a curb candidate approximation line formed by fitting the plurality of candidate points associated with the further two-dimensional plane, and when the curb candidate approximation line is associated with the two-dimensional plane intersecting in the vehicle length direction, if the curb candidate approximation line is included in the target range and the current position of the vehicle lies on the curb candidate approximation line, the device determines to stop the recognition of the object, and if the curb candidate approximation line is included in the target range and the current position of the vehicle lies on the curb candidate approximation line, the device determines not to stop the recognition of the object.

7. A determination method for determining a convex shape on a snowy road in front of a vehicle, comprising: a computer acquiring a stereo image including an object in front of the vehicle; generating a disparity image capable of determining the distance to the object based on the stereo image; associating a plurality of candidate points corresponding to the object with a two-dimensional plane intersecting the vehicle's length direction based on the disparity image; and determining, based on the plurality of candidate points, whether or not a convex shape protruding in the vehicle's height direction exists within a target range on both the left and right sides of the two-dimensional plane along the vehicle's width from the center of the vehicle's width.

8. A non-temporary tangible recording medium on which a determination program for determining a convex shape on a snowy road in front of a vehicle is recorded, wherein the determination program is recorded which causes a computer to: acquire a stereo image including an object in front of the vehicle; generate a parallax image capable of determining the distance to the object based on the stereo image; associate a plurality of candidate points corresponding to the object with a two-dimensional plane intersecting the vehicle's length direction based on the parallax image; and determine whether or not a convex shape protruding in the vehicle's height direction exists in a target area on both the left and right sides of the two-dimensional plane along the vehicle's width from the center of the vehicle's width, based on the plurality of candidate points.

Images