Environment recognition device and recording medium

Abstract

This environment recognition device executes detection of a three-dimensional object on the basis of an image acquired by an imaging device, and detection of a curb and a side wall on the basis of the image. When detecting the curb and the side wall, the environment recognition device obtains a plurality of detection points by projecting, onto a two-dimensional plane composed of a lateral direction and a depth direction that are viewed from the imaging device, unit blocks having heights from the road surface in a predetermined range among unit blocks, each being composed of a predetermined number of pixels as a processing unit in the image, and determines whether the three-dimensional object detected by the processing of detecting the three-dimensional object is a vehicle, and when the three-dimensional object is a vehicle, the environment recognition device generates information on the detection results of the curb and the side wall on the basis of the detection points other than the detection points indicating the vehicle among the plurality of detection points.

Description

Environmental recognition device and recording medium

[0001] The present disclosure relates to an environmental recognition device and a recording medium.

[0002] In recent advanced driving assistance systems for vehicles, environmental recognition technologies that utilize imaging devices that generate frame images, such as stereo cameras, LiDAR (Light Detection And Ranging), or millimeter-wave radars, have been used. For example, Patent Document 1 discloses a technique for detecting a curb using point cloud information obtained by a distance sensor mounted on a vehicle.

[0003] Japanese Patent Application Laid-Open No. 2022-178758

[0004] When detecting a curb or a side wall from a frame image acquired by an imaging device, an object having a predetermined height on the road surface can be identified to find an object that seems to be a curb or a side wall. However, since vehicles are also included in the objects having a predetermined height on the road surface, there is a risk of misdetecting a vehicle as a curb or a side wall.

[0005] The present disclosure has been made in view of the above problems, and aims to suppress misdetection of a curb or a side wall and stabilize the detection result.

[0006] In order to solve the above problems, according to an aspect of the present disclosure, there is provided an environmental recognition device including a solid object detection processing unit that detects a solid object based on an image acquired by an imaging device, and a curb and side wall detection processing unit that detects a curb and a side wall based on the image. The curb and side wall detection processing unit projects a unit block having a height within a predetermined range from the road surface among unit blocks each composed of a predetermined number of pixels that are processing units in the image onto a two-dimensional plane composed of the horizontal and depth directions as viewed from the imaging device to obtain a plurality of detection points, determines whether the solid object detected by the solid object detection processing unit is a vehicle, and when the solid object is the vehicle, generates information on the detection results of the curb and the side wall based on the remaining detection points excluding the detection points indicating the vehicle from the plurality of detection points.

[0007] Furthermore, in order to solve the above problems, according to another aspect of this disclosure, a non-temporary tangible recording medium is provided which records a program that causes one or more processors to perform the following: to detect a three-dimensional object based on an image acquired by an imaging device, and to detect curbs and side walls based on the image; to determine whether the three-dimensional object detected by the three-dimensional object detection processing unit is a vehicle, and if the three-dimensional object is a vehicle, to generate information on the detection result of the curbs and side walls based on the remaining detection points obtained by excluding the detection point indicating the vehicle from the plurality of detection points.

[0008] As explained above, this disclosure makes it possible to suppress false detections of curbs or side walls and stabilize the detection results.

[0009] This is a schematic diagram showing an example configuration of a vehicle equipped with an environmental recognition device according to the present disclosure. This is a block diagram showing an example configuration of an environmental recognition device according to the same embodiment. This is a flowchart of the curb side wall detection process performed by the environmental recognition device according to the same embodiment. This is an explanatory diagram showing the process of obtaining a plurality of detection points projected onto the xz two-dimensional plane by the environmental recognition device according to the same embodiment. This is an explanatory diagram showing the process of obtaining a plurality of detection points projected onto the xz two-dimensional plane by the environmental recognition device according to the same embodiment. This is an explanatory diagram showing an example of a group of curb candidate points obtained by the environmental recognition device according to the same embodiment. This is an explanatory diagram showing the process of excluding vehicle candidate points from a group of curb candidate points by the environmental recognition device according to the same embodiment. This is an explanatory diagram showing the process of excluding vehicle candidate points from a group of curb candidate points by the environmental recognition device according to the same embodiment. This is an explanatory diagram showing the process of excluding detection points (vehicle candidate points) representing a line of cars from a group of curb candidate points by the environmental recognition device according to the same embodiment. This is an explanatory diagram showing the process of excluding detection points (vehicle candidate points) representing a line of cars from a group of curb candidate points by the environmental recognition device according to the same embodiment. This is an explanatory diagram illustrating the process of excluding detection points (vehicle candidate points) representing a line of vehicles from a group of curb candidate points by the environmental recognition device according to the same embodiment. This is an explanatory diagram illustrating the process of excluding detection points (vehicle candidate points) representing a line of vehicles from a group of curb candidate points by the environmental recognition device according to the same embodiment.

[0010] Preferred embodiments of this disclosure will be described in detail below with reference to the attached drawings. The specific dimensions, materials, numerical values, etc., shown in the following embodiments are merely examples to facilitate understanding of the invention and do not limit the present invention unless otherwise specified. In this specification and drawings, components having substantially the same functional configuration are denoted by the same reference numerals to avoid redundant explanations.

[0011] <1. Overall Vehicle Configuration> First, an example of the overall configuration of a vehicle equipped with the environmental recognition device according to the embodiment of this disclosure will be described.

[0012] Figure 1 is a schematic diagram showing an example of the configuration of a vehicle 1 equipped with an environmental recognition device 50 according to this embodiment. The vehicle 1 shown in Figure 1 is configured as a four-wheel drive vehicle that transmits the drive torque output from the drive source 9 to the left front wheel 3FL, the right front wheel 3FR, the left rear wheel 3RL, and the right rear wheel 3RR (hereinafter collectively referred to as "wheels 3" unless otherwise specified). The drive source 9 may be an internal combustion engine such as a gasoline engine or a diesel engine, a drive motor, or both.

[0013] Vehicle 1 may be an electric vehicle equipped with two drive motors, for example, a front-wheel drive motor and a rear-wheel drive motor, or an electric vehicle equipped with a drive motor corresponding to each wheel 3. Vehicle 1 may also be a two-wheel drive vehicle with front-wheel drive or rear-wheel drive. Furthermore, if vehicle 1 is an electric vehicle or a hybrid electric vehicle, vehicle 1 may be equipped with a secondary battery that stores the power supplied to the drive motors, or a generator such as a motor or fuel cell that generates the power charged to the battery.

[0014] Vehicle 1 is equipped with a drive source 9, an electric steering device 15, and brake devices 17FL, 17FR, 17RL, and 17RR (hereinafter collectively referred to as "brake device 17" unless otherwise specified) as equipment used for controlling the operation of Vehicle 1. The drive source 9 outputs drive torque that is transmitted to the front wheel drive shaft 5F and the rear wheel drive shaft 5R via a transmission (not shown), a front wheel differential mechanism 7F, and a rear wheel differential mechanism 7R. The drive of the drive source 9 and the transmission is controlled by a control device 40 which is configured to include one or more electronic control units (ECUs).

[0015] The electric steering system 15 includes an electric motor and gear mechanism (not shown) and is controlled by the control device 40 to adjust the steering angles of the left front wheel 3FL and the right front wheel 3FR. The control device 40 controls the electric steering system 15 so that the vehicle 1 travels in a predetermined position when the driver assistance function switch is turned on and the driving mode is set to the driver assistance mode. The control device 40 also controls the electric steering system 15 based on the steering angle of the steering wheel 13 by the driver when the driving mode is set to the manual driving mode.

[0016] The braking system 17 applies braking force to each of the front, rear, left, and right wheels 3. The braking system 17 is configured, for example, as a hydraulic braking system. A hydraulic control unit 41 controls the hydraulic pressure supplied to each of the braking systems 17, thereby generating a predetermined braking force. If the vehicle 1 is an electric vehicle or a hybrid electric vehicle, the braking system 17 is used in conjunction with regenerative braking by the drive motor.

[0017] The control device 40 includes one or more electronic control devices that control the drive of the power source 9, the electric steering device 15, and the hydraulic control unit 41. The control device 40 may also have a function to control the drive of the transmission that changes the speed of the output from the power source 9 and transmits it to the wheels 3. The control device 40 is configured to perform driving assistance processing for the vehicle 1 using information on obstacles recognized by the environment recognition device 50. For example, the control device 40 performs lane keeping control, which controls the electric steering device 15 so that the vehicle 1 does not deviate from the boundary line of the driving lane. However, the driving assistance processing is not limited to lane keeping control, and may be driving assistance processing by conventionally known advanced driving assistance functions.

[0018] Vehicle 1 is also equipped with a pair of left and right stereo cameras 31FL and 31FR, a vehicle status sensor 35, a GNSS (Global Navigation Satellite System) antenna 37, a switch 39, a notification device 43, and an environmental recognition device 50. The vehicle status sensor 35, GNSS antenna 37, switch 39, notification device 43, and environmental recognition device 50 are connected to the control device 40 via a dedicated line or communication means such as CAN (Controller Area Network) or LIN (Local Internet).

[0019] The stereo cameras 31FL and 31FR are an embodiment of an imaging device that photographs the area in front of the vehicle 1 and generates a pair of stereo images (captured images). The stereo cameras 31FL and 31FR are imaging devices equipped with an image sensor such as a CCD (Charged-Coupled Device) or CMOS (Complementary Metal-Oxide-Semiconductor). The stereo cameras 31FL and FR are connected to the environment recognition device 50 via wired or wireless communication means and transmit the generated stereo images to the environment recognition device 50.

[0020] In addition to the stereo cameras 31FL and 31FR, vehicle 1 may also be equipped with a camera that photographs the rear of vehicle 1, or a camera mounted on a side mirror or the like that photographs the left rear or right rear. Furthermore, in addition to the stereo cameras 31FL and 31FR, vehicle 1 may also be equipped with an imaging device that generates frame images including measurement point clouds such as LiDAR or millimeter-wave radar.

[0021] The vehicle state sensor 35 consists of one or more sensors that detect the operating state and behavior of the vehicle 1. The vehicle state sensor 35 includes, for example, a steering angle sensor, an accelerator position sensor, a brake stroke sensor, and a brake pressure sensor. It also includes, for example, a vehicle speed sensor, an acceleration sensor, and an angular velocity sensor. The vehicle state sensor 35 transmits a sensor signal indicating the detected information to the control device 40.

[0022] The GNSS antenna 37 receives satellite signals transmitted from satellites such as GPS (Global Positioning System). The GNSS antenna 37 transmits the vehicle 1's position information, which is included in the received satellite signals, to the control device 40.

[0023] Switch 39 is operated by the driver to switch on or off the driver assistance function, which is a system that takes over the driving task of the vehicle 1. Switch 39 may be a physical switch, a touch panel, or a voice input device.

[0024] The environmental recognition device 50 comprises one or more processors such as CPUs (Central Processing Units) and one or more memories connected to the one or more processors in a communicative manner. The environmental recognition device 50 functions as a device that performs processing to recognize the environment around the vehicle 1 by having one or more processors execute a computer program. The computer program is a computer program that causes the processor to execute the operations to be performed by the environmental recognition device 50, which will be described later. The computer program executed by the processor may be recorded on a recording medium that functions as a memory unit provided in the environmental recognition device 50, or it may be recorded on a recording medium built into the environmental recognition device 50 or on any external recording medium that can be attached to the environmental recognition device 50.

[0025] Recording media for storing computer programs may include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs, DVDs, and Blu-ray®; magneto-optical media such as floppy disks; memory elements such as RAM (Random Access Memory) or ROM (Read Only Memory); flash memory such as USB memory and SSDs; and other media capable of storing programs.

[0026] The environmental recognition device 50 performs a process to recognize the environment around the vehicle 1 based on the data transmitted from the stereo cameras 31FL and 31FR. The environmental recognition device 50 performs a process to detect three-dimensional objects based on images acquired by the stereo cameras 31FL and 31FR, and a process to detect curbs and side walls based on images acquired by the stereo cameras 31FL and 31FR. The environmental recognition device 50 outputs the recognized information about the surrounding environment to the control device 40.

[0027] The notification device 43 is driven by the control device 40 and notifies the occupants of various information by means of image display, audio output, etc. The notification device 43 includes, for example, a display device provided in the instrument panel and a speaker provided in the vehicle 1. The display device may be a display device of a navigation system.

[0028] <2. Environmental Recognition Device> Next, the environmental recognition device 50 according to this embodiment will be described in detail.

[0029] (2-1. Configuration Example) Figure 2 is a block diagram showing the configuration of the part of the environmental recognition device 50 according to this embodiment that is related to the process of detecting curbs and side walls. The environmental recognition device 50 is connected to stereo cameras 31FL, 31FR (hereinafter referred to as stereo camera 31) via a dedicated line or a communication means such as CAN or LIN. The environmental recognition device 50 is also connected to a control device 40 via a dedicated line or a communication means such as CAN or LIN.

[0030] The environmental recognition device 50 comprises a processing unit 51 and a storage unit 53. The processing unit 51 is configured with one or more processors such as CPUs. Part or all of the processing unit 51 may be configured with updatable components such as firmware, or it may be a program module executed by commands from the CPU or the like.

[0031] The storage unit 53 is composed of one or more memory elements (RAM and ROM) that are connected to the processing unit 51 in a communicative manner. However, the number and type of storage units 53 are not particularly limited. The storage unit 53 stores computer programs executed by the processing unit 51, various parameters used in arithmetic processing, detection data, calculation results, and other data. A portion of the storage unit 53 is used as the work area of ​​the processing unit 51.

[0032] In addition, the environmental recognition device 50 is equipped with an interface (not shown) for transmitting and receiving data with the stereo cameras 31FL, 31FR and the control device 40.

[0033] The processing unit 51 includes a position derivation processing unit 61, a three-dimensional object detection processing unit 63, and a curb side wall detection processing unit 65. The position derivation processing unit 61, the three-dimensional object detection processing unit 63, and the curb side wall detection processing unit 65 are functions realized by the execution of a program by one or more processors. Note that some or all of the position derivation processing unit 61, the three-dimensional object detection processing unit 63, and the curb side wall detection processing unit 65 may be composed of hardware such as analog circuits.

[0034] The functions of the position derivation processing unit 61, the three-dimensional object detection processing unit 63, and the curb side wall detection processing unit 65 will be briefly explained below, followed by a detailed explanation of the specific processing operations.

[0035] (Position Derivation Processing Unit) The position derivation processing unit 61 performs a process to derive the three-dimensional position in real space for each unit block consisting of a predetermined number of pixels, which serves as the processing unit, from a pair of stereo images output from the stereo camera 31. A unit block is represented, for example, by an array of 4 horizontal pixels × 4 vertical pixels, but it may be set to any number of pixels. The position derivation processing unit 61 generates a distance image that associates the derived three-dimensional position with each unit block.

[0036] (3D Object Detection Processing Unit) The 3D object detection processing unit 63 performs a process to detect 3D objects present around the vehicle 1 based on the 3D position information for each unit block generated by the position derivation processing unit 61. For example, the 3D object detection processing unit 63 identifies 3D objects by grouping blocks that are located above the road surface, have the same color value, and whose 3D position difference in the distance image is within a predetermined range. Specifically, the 3D object detection processing unit 63 groups blocks whose x-coordinate difference, y-coordinate difference, and z-coordinate difference in the distance image are within a predetermined range (for example, 0.1 m) assuming that they correspond to the same 3D object.

[0037] The 3D object detection processing unit 63 records the detected 3D object information along with information on the object's speed, position, height, and width. The speed of the 3D object can be calculated based on the distance traveled on the three-dimensional coordinate system and the speed of the vehicle 1 for each calculation cycle. The position, height, and width of the 3D object can be calculated based on its position and length on the three-dimensional coordinate system.

[0038] (Curb and Sidewall Detection Processing Unit) The curb and sidewall detection processing unit 65 performs the process of detecting curbs and sidewalls based on the three-dimensional position information for each unit block generated by the position derivation processing unit 61. In this embodiment, the curb and sidewall detection processing unit 65 performs the following: obtain a plurality of detection points by projecting unit blocks in the captured image whose height from the road surface is within a predetermined range onto a two-dimensional plane consisting of the horizontal direction (x-axis) and depth direction (z-axis) as seen from the stereo camera 31; determine whether the three-dimensional object detected by the three-dimensional object detection processing unit 63 is a vehicle; and, if the three-dimensional object is a vehicle, generate information on the detection results of curbs and sidewalls based on the remaining detection points after excluding the detection points indicating the vehicle from the plurality of detection points. The curb and sidewall detection processing unit 65 outputs the obtained curb and sidewall detection results to the control device 40. The control device 40 performs the process of supporting the driving of the vehicle 1 using the acquired curb and sidewall detection information.

[0039] (2-2. Operation) Next, the processing operation of the environmental recognition device 50 according to this embodiment will be described in detail with reference to the drawings as appropriate. In the following description, the detection target of the curb side wall detection processing unit 65 will be referred to as "curb," but this "curb" does not refer only to curbs; the detection target can also be a side wall.

[0040] Figure 3 shows a flowchart of the curb wall detection process performed by the environmental recognition device 50. When the system of the vehicle 1, including the environmental recognition device 50 and the control device 40, is started (step S11), the position derivation processing unit 61 of the processing unit 51 acquires a pair of stereo images from the stereo camera 31 and performs position derivation processing (step S13). For example, the position derivation processing unit 61 uses pattern matching technology to derive parallax information, including the parallax and the image position indicating the position of an arbitrary unit block within the image. Then, the position derivation processing unit 61 searches the other image for a block corresponding to a unit block arbitrarily extracted from one of the pair of stereo images. A unit block is, for example, an area of ​​4 pixels horizontally x 4 pixels vertically, but the size of the unit block is not limited to the above example. The position derivation processing unit 61 performs the process of deriving parallax information for each unit block for all blocks of the detection area set in the image. The position derivation processing unit 61 generates a distance image that associates the derived parallax information with each unit block.

[0041] The position derivation processing unit 61 converts the disparity information for each derived unit block into a position (three-dimensional position) on a three-dimensional coordinate system in real space, where the horizontal direction (vehicle width direction) is the x-axis, the height direction is the y-axis, and the depth direction is the z-axis, according to the principle of triangulation. At this time, the position derivation processing unit 61 derives the height position (y-coordinate) of each block from the road surface based on the z-coordinate of each block and the distance in the image between the block and a point on the road surface at the same z-coordinate as the block. Then, the position derivation processing unit 61 stores the derived three-dimensional position in association with the distance image.

[0042] The process of deriving disparity information and calculating the three-dimensional position by the position derivation processing unit 61 is merely an example, and may be carried out using various known techniques.

[0043] Next, the curb side wall detection processing unit 65 obtains a plurality of detection points projected onto a two-dimensional plane composed of the lateral direction (hereinafter also referred to as the "x direction") and the depth direction (hereinafter also referred to as the "z direction") as viewed from the stereo camera 31 for unit blocks in the distance image whose height from the road surface is within a predetermined range (step S15).

[0044] FIGS. 4 to 5 show an example of a process for obtaining a plurality of detection points obtained by projecting unit blocks whose height from the road surface is within a predetermined range onto the xz two-dimensional plane. FIG. 4 shows a state in which the vehicle (own vehicle) 1 travels in the left lane of a two-lane road on one side, the other vehicle 73 travels in the right lane, and a curb 71 exists on the left side of the road.

[0045] For example, when the curb side wall detection processing unit 65 scans the information on the three-dimensional position of the unit block along the x direction in the distance image, the unit block at the change point where the height from the road surface exceeds zero from zero is specified. That is, the curb side wall detection processing unit 65 specifies the unit block where a height difference appears on the road surface in the lateral direction as viewed from the vehicle 1. The curb side wall detection processing unit 65 repeats the process of specifying the unit block where the above height difference appears from the front side to the back side as viewed from the vehicle 1. In the example shown in FIG. 4, the unit blocks corresponding to the positions of the side surfaces of the curb 71 and the other vehicle 73 are specified.

[0046] Note that when the curb side wall detection processing unit 65 scans the information on the three-dimensional position of the unit block along the x direction, it may scan from the left end to the right end, or from the right end to the left end, or may scan in both the left and right directions along the x direction from the front of the vehicle 1 (the position where x = 0 in FIG. 5), or may scan from the left end and the right end toward the front direction of the vehicle 1. Also, the curb side wall detection processing unit 65 may repeat the process of specifying the unit block where the above height difference appears from the back side to the front side as viewed from the vehicle 1, and the order thereof is not particularly limited. The specified unit blocks are all unit blocks at the change points where the height from the road surface exceeds zero from zero, but at least one of the upper limit and the lower limit of the height from the road surface may be set to any appropriate value.

[0047] Then, as shown in FIG. 5, the curb sidewall detection processing unit 65 maps the detection points obtained by projecting the specified unit blocks on the xz two-dimensional plane. For example, the xyz three-dimensional space on the xz two-dimensional plane with the x direction being 31,242 mm and the z direction being 180,224 mm is set as the detection range, and the detection points obtained by projecting the unit blocks where the above height difference appears within the detection range are mapped on the xz two-dimensional plane.

[0048] Next, the curb sidewall detection processing unit 65 calculates candidate points (curb candidate points) indicating the curb among the plurality of detection points (step S17). The process of calculating the curb candidate points from the plurality of detection points may be executed by any method. For example, the curb sidewall detection processing unit 65 groups the detection points with close distances between adjacent detection points according to a given criterion. Note that the entire range of the xz two-dimensional plane is referred to as the "search range", and the regions on both the left and right sides of the vehicle excluding the front region of the vehicle are also referred to as the "valid range".

[0049] Next, the curb sidewall detection processing unit 65 obtains a quadratic approximation model for the grouped plurality of detection points. For example, the curb sidewall detection processing unit 65 calculates the quadratic approximation model by a so-called RANSAC method. Then, the curb sidewall detection processing unit 65 extracts the detection points plotted on the quadratic approximation model among the plurality of detection points. The curb sidewall detection processing unit 65 performs the calculation of the quadratic approximation model and the extraction of the detection points for each group of the grouped detection points. Note that the method for calculating the quadratic approximation model is not particularly limited. Thereby, the curb candidate points are obtained from the stereo images acquired in each operation cycle.

[0050] FIG. 6 shows a group of curb candidate points obtained from the detection points shown in FIG. 5. In the example shown in FIG. 6, two groups 81 and 83 of curb candidate points are specified.

[0051] Note that the process of specifying the above-described curb candidate points is merely an example, and the curb sidewall detection processing unit 65 may specify the curb candidate points by other methods. Also, the curb sidewall detection processing unit 65 may execute the process of specifying the curb candidate points and the process of specifying the candidate points of the sidewall by different methods.

[0052] Returning to the flowchart in Figure 3, the curb wall detection processing unit 65 then identifies vehicle candidate points, which are detection points representing vehicles and convoys, from among the group of curb candidate points (step S19). Based on the information on the speed, position, height, and width of the three-dimensional object detected by the three-dimensional object detection processing unit 63, and matching information for identifying vehicles, the curb wall detection processing unit 65 determines whether or not the three-dimensional object detected by the three-dimensional object detection processing unit 63 is a vehicle. The matching information is, for example, a block pattern that characterizes the grouped block pattern in the distance image as being a vehicle.

[0053] For example, the curb side wall detection processing unit 65 determines whether a three-dimensional object is a vehicle based on the speed of the three-dimensional object, the length of the three-dimensional object in the z direction, the distance from vehicle 1 to the three-dimensional object in the z direction, the distance from vehicle 1 to the three-dimensional object in the x direction, and matching information that identifies whether the three-dimensional object is a vehicle traveling in an adjacent lane or an oncoming vehicle traveling in an oncoming lane. The condition for the speed of the three-dimensional object is set to any appropriate range depending on whether the three-dimensional object is a vehicle traveling in a parallel direction or an oncoming vehicle. The conditions for the length of the three-dimensional object in the z direction, the distance from vehicle 1 to the three-dimensional object in the z direction, and the distance from vehicle 1 to the three-dimensional object in the x direction are set to any appropriate range that can be assumed when the three-dimensional object is assumed to be a vehicle.

[0054] Next, if the curb side wall detection processing unit 65 determines that the three-dimensional object is a vehicle, it excludes vehicle candidate points, which are detection points representing a vehicle, from the multiple detection points (step S21). The curb side wall detection processing unit 65 identifies a region on the xz two-dimensional plane that projects the side and front or rear parts visible from the vehicle 1 of the three-dimensional object determined to be a vehicle, and excludes vehicle candidate points located in that region from the group of curb candidate points. The three-dimensional object information obtained from the stereo image is shown by the surface of the three-dimensional object visible from the stereo camera 31, and the region obtained by projecting this three-dimensional object information onto the xz two-dimensional plane represents the outline of the three-dimensional object.

[0055] Figures 7 and 8 show the process of excluding vehicle candidate points from the group of curb candidate points in the example shown in Figures 4 to 6. As shown in Figure 7, the curb side wall detection processing unit 65 identifies the region obtained by projecting the side and rear of the other vehicle 73 onto the xz two-dimensional plane from the three-dimensional object information 85, 86, and 87 detected by the three-dimensional object detection processing unit 63. Of the groups of curb candidate points 81 and 83 (see Figure 6) obtained in step S17, group 83 of curb candidate points that exist in that region corresponds to the detected point (vehicle candidate point) representing the other vehicle 73. Then, as shown in Figure 8, the curb side wall detection processing unit 65 excludes vehicle candidate points from groups 81 and 83 of curb candidate points, leaving only group 81 of curb candidate points that represent curbs.

[0056] Furthermore, after identifying a three-dimensional object representing a vehicle, the curb side wall detection processing unit 65 determines that if other three-dimensional objects exist within a predetermined position and distance range from the three-dimensional object determined to be a vehicle, it determines both the three-dimensional object determined to be a vehicle and the other three-dimensional objects to be a convoy of vehicles, and further excludes detection points due to the convoy of vehicles.

[0057] Figures 9 to 12 show the process of further determining the convoy of vehicles and excluding detection points (vehicle candidate points) representing the convoy from the group of curb candidate points. As shown in Figure 9, vehicle (own vehicle) 1 is traveling in the left lane of a two-lane road, another vehicle 73 is traveling in the right lane, and a preceding vehicle 75 is traveling in front of them. As shown in Figure 10, the curb side wall detection processing unit 65 maps detection points, which are unit blocks of change points where the height from the road surface changes from zero to above zero, onto the xz two-dimensional plane, and identifies three groups of curb candidate points 81, 83, and 84.

[0058] Next, the curb side wall detection processing unit 65 determines whether the object detected by the object detection processing unit 63 is a vehicle, based on the information regarding the speed, position, height, and width of the object detected by the object detection processing unit 63, and matching information for identifying a vehicle. Here, since the side and rear of the other vehicle 73 are visible from the stereo camera 31 of vehicle 1, the object representing the other vehicle 73 can be identified. On the other hand, although part or all of the side of the preceding vehicle 75 is visible from the stereo camera 31 of vehicle 1, depending on the distance between the other vehicle 73 and the preceding vehicle 75 and the position of vehicle 1, part or all of the rear of the preceding vehicle 75 may not be visible. For this reason, in the process of determining whether or not the object is a vehicle as described above, it may not be possible to identify the object representing the preceding vehicle 75 based on the stereo image.

[0059] Therefore, if the curb side wall detection processing unit 65 determines that there is other three-dimensional object information 88 within an area R specified by a predetermined position and distance range for the three-dimensional object information 87 which has been determined to be another vehicle 73, it determines that the three-dimensional object information 87 which has been determined to be another vehicle 73 and the other three-dimensional object information 88 are three-dimensional object information 87 and 88 which represent a train of vehicles. The process of determining the train of vehicles is performed on three-dimensional object information that has at least side information, and not on three-dimensional object information that does not have side information. In the example shown in Figure 11, among the groups 81, 83, and 84 of curb candidate points shown in Figure 10, group 83 of curb candidate points that are located in the region obtained by projecting the side of the other vehicle 73 onto the xz two-dimensional plane corresponds to a detection point (vehicle candidate point) that represents the other vehicle 73.

[0060] Furthermore, when the curb side wall detection processing unit 65 projects the three-dimensional object information onto the xz two-dimensional plane, it determines that three-dimensional object information 88 whose side information exists within an area R, which is specified based on the side of another vehicle 73 in the x and z directions, respectively, is a vehicle (leading vehicle 75), even if there is no front or rear information, and determines that it represents a convoy of vehicles. When the curb side wall detection processing unit 65 identifies other three-dimensional object information 88 that make up the convoy of vehicles for the three-dimensional object information 87 identified as another vehicle 73, it repeatedly determines whether there is other three-dimensional object information included in the convoy based on the side of the three-dimensional object information 88. The range of area R for determining the convoy of vehicles may be set to any appropriate value.

[0061] Then, as shown in Figure 12, the curb side wall detection processing unit 65 excludes the vehicle candidate points of the three-dimensional object information 87, 88 representing the convoy from the curb candidate point groups 81, 83, 84, leaving only the curb candidate point group 81 representing the curb. In this way, the three-dimensional object information of the convoy, which consists of multiple vehicles, can be identified, and only the group of curb candidate points excluding the vehicle candidate points can be identified.

[0062] In the process of excluding vehicle candidate points representing a convoy from the curb candidate points described above, the curb side wall detection processing unit 65 identified three-dimensional object information 87 and 88 representing a convoy, and then excluded groups 83 and 84 corresponding to vehicle candidate points of the three-dimensional object information 87 and 88. However, if, after identifying group 83 of curb candidate points corresponding to three-dimensional object information 87 representing another vehicle 73, there is a group 84 of other curb candidate points within an area R identified at a predetermined position and distance range relative to group 83, then the curb side wall detection processing unit 65 may identify these groups 83 and 84 of curb candidate points as vehicle candidate points representing a convoy and exclude them from the curb candidate points.

[0063] Next, the curb wall detection processing unit 65 generates curb detection result information (output information) based on the remaining group of curb candidate points after excluding vehicle candidate points (step S23). The output information of the curb detection result may be information of the point cloud of the group of curb candidate points, but in this embodiment, the curb wall detection processing unit 65 generates information of a quadratic approximation of the group of curb candidate points as output information. The quadratic approximation can be shown, for example, by the following equation (1): x = az ２ +bz+c …(1) a: curvature b: yaw angle relative to the driving lane c: lateral position relative to the driving lane

[0064] In this case, the curb wall detection processing unit 65 may perform noise reduction processing or correction processing to further stabilize the detection results for use in driving assistance processing. The curb wall detection processing unit 65 generates information of parameters a, b, and c that specify the quadratic approximation in the above equation (1) as output information.

[0065] Next, the curb wall detection processing unit 65 generates output information for a quadratic approximation of the group of curb candidate points, and then transmits this output information to the control device 40 (step S25). The control device 40, having received the output information, executes driving support processing using the detected curb and wall information. The curb wall detection processing unit 65 may also determine whether to execute or cancel the driving support processing using the output information from the control device 40 based on the reliability of the detection results, and may include this determination result in the output information before transmitting it. This prevents the control device 40, upon receiving the output information, from executing driving support processing using incorrect curb and wall information.

[0066] Next, the curb wall detection processing unit 65 determines whether or not the system of the vehicle 1, including the environmental recognition device 50 and the control device 40, will stop (step S27). If the system does not stop (S27 / No), the process returns to step S13, and the processing of each of the steps described above is repeated as the next calculation cycle. On the other hand, if the system does stop (S27 / Yes), the curb wall detection processing unit 65 terminates the process of detecting the curb and side wall.

[0067] As described above, the environmental recognition device 50 according to this embodiment includes a three-dimensional object detection processing unit 63 that detects three-dimensional objects based on stereo images acquired by a stereo camera 31, and a curb / side wall detection processing unit 65 that detects curbs and side walls based on stereo images. The curb / side wall detection processing unit 65 has the following configuration: it determines a plurality of detection points by projecting unit blocks consisting of a predetermined number of pixels, which are processing units in the stereo image, with a predetermined height range from the road surface onto a two-dimensional xz plane consisting of the horizontal and depth directions as viewed from the stereo camera 31; it determines whether the three-dimensional object detected by the three-dimensional object detection processing unit 63 is a vehicle; and, if the three-dimensional object is a vehicle, it generates information on the detection results of curbs and side walls based on the remaining detection points after excluding the detection points indicating the vehicle from the plurality of detection points. This reduces the risk of mistakenly identifying detection points by vehicles with sides along the driving lane as curbs or side walls, and stabilizes the detection results of curbs and curbs. Therefore, it is possible to prevent driving support processes such as lane keeping control from being executed based on incorrect detection information.

[0068] Furthermore, in the environmental recognition device 50 according to this embodiment, if the three-dimensional object is a vehicle, the curb and side wall detection processing unit 65 identifies an area visible from the stereo camera 31 on the vehicle and generates information on the detection result of curbs and side walls based on the remaining detection points after excluding the detection points in that area from the detection point group. This prevents the device from mistakenly identifying detection points from the side and front or rear of other vehicles as curbs or side walls.

[0069] Furthermore, in the environmental recognition device 50 according to this embodiment, the curb side wall detection processing unit 65 determines whether the three-dimensional object detected by the three-dimensional object detection processing unit 63 is a vehicle, based on at least one of the speed, location, height, width, and matching information for identifying a vehicle of the three-dimensional object detected by the three-dimensional object detection processing unit 63. This improves the accuracy of determining whether a three-dimensional object is a vehicle and prevents the detection points of the side and front or rear of other vehicles from being mistakenly identified as curbs or side walls.

[0070] Furthermore, in the environmental recognition device 50 according to this embodiment, if there are other three-dimensional objects within a predetermined position and distance range relative to a three-dimensional object determined to be a vehicle, the curb and side wall detection processing unit 65 determines the three-dimensional object and the other three-dimensional objects to be a convoy of vehicles, and generates information on the detection results of curbs and side walls based on the remaining detection points after further excluding the detection points of the convoy. This prevents the detection points of a group of vehicles from being mistakenly identified as curbs or side walls, even if some of the vehicles in the convoy are in a blind spot due to other vehicles and there are three-dimensional objects that cannot be determined to be vehicles.

[0071] While preferred embodiments of the present disclosure have been described in detail above with reference to the attached drawings, the present disclosure is not limited to such examples. It is clear to any person with ordinary skill in the art to which the present disclosure pertains that various modifications or alterations may be conceived within the scope of the technical idea set forth in the claims, and these will naturally also be understood to fall within the technical scope of the present disclosure.

[0072] Furthermore, the technology disclosed herein can also be realized as a vehicle equipped with the environmental recognition device described in the embodiments above, an environmental recognition processing method executed by the environmental recognition device, a computer program that causes a computer to function as the environmental recognition device described above, and a non-temporary tangible recording medium on which the computer program is recorded.

[0073] 1: Vehicle (own vehicle) 31, 31FL, 31FR: Stereo camera 50: Environmental recognition device 51: Processing unit 53: Memory unit 61: Position derivation processing unit 63: Three-dimensional object detection processing unit 65: Curb side wall detection processing unit 71: Curb 73: Vehicle (other vehicle) 75: Preceding vehicle

Claims

1. An environmental recognition device comprising: a three-dimensional object detection processing unit that detects three-dimensional objects based on an image acquired by an imaging device; and a curb and side wall detection processing unit that detects curbs and side walls based on the image, wherein the curb and side wall detection processing unit performs the following: to obtain a plurality of detection points by projecting unit blocks consisting of a predetermined number of pixels that serve as processing units in the image, where the height from the road surface is within a predetermined range, onto a two-dimensional plane consisting of the horizontal and depth directions as viewed from the imaging device; to determine whether the three-dimensional object detected by the three-dimensional object detection processing unit is a vehicle; and, if the three-dimensional object is a vehicle, to generate information on the detection results of the curb and the side wall based on the remaining detection points after excluding the detection point indicating the vehicle from the plurality of detection points.

2. The environmental recognition device according to claim 1, wherein, when the curb side wall detection processing unit is the vehicle, it identifies an area visible from the imaging device on the vehicle and generates information on the detection results of the curb and the side wall based on the remaining detection points obtained by excluding the detection points in the area from the detection points.

3. The environmental recognition device according to claim 1, wherein the curb side wall detection processing unit determines whether the three-dimensional object detected by the three-dimensional object detection processing unit is the vehicle, based on at least one of the speed, location, height, width, and matching information for identifying the vehicle of the three-dimensional object detected by the three-dimensional object detection processing unit.

4. The environmental recognition device according to claim 1, wherein the curb side wall detection processing unit determines that if other three-dimensional objects exist within a predetermined position and distance range from the three-dimensional object determined to be a vehicle, the three-dimensional object and the other three-dimensional objects are a line of vehicles, and generates information on the detection results of the curb and the side wall based on the remaining detection points after further excluding the detection points of the line of vehicles.

5. A non-temporary tangible recording medium that records a program causing one or more processors to perform the following: detect a three-dimensional object based on an image acquired by an imaging device; detect a curb and a side wall based on the image; and, in detecting the curb and the side wall, to determine a plurality of detection points by projecting unit blocks consisting of a predetermined number of pixels that constitute the processing unit in the image, where the height from the road surface is within a predetermined range, onto a two-dimensional plane consisting of the horizontal and depth directions as viewed from the imaging device; determine whether the three-dimensional object detected by the process of detecting the three-dimensional object is a vehicle; and, if the three-dimensional object is a vehicle, generate information on the detection result of the curb and the side wall based on the remaining detection points after excluding the detection point indicating the vehicle from the plurality of detection points.

Images