Environment recognition device and recording medium

Abstract

This environment recognition device comprises: a three-dimensional object detection processing unit that detects a three-dimensional object at least on the basis of an image captured by a stereo camera; and an intermittent wall detection processing unit that detects an intermittent wall on the basis of the captured image. The intermittent wall detection processing unit: obtains a plurality of detection points obtained by projecting, onto a two-dimensional plane formed of the lateral direction and the depth direction as viewed from the stereo camera, a unit block of which the height from a road surface is in a prescribed range from among unit blocks each formed of a prescribed number of pixels that is the unit for which processing is performed in the captured image; obtains a quadratic approximation model of the plurality of detection points; creates intermittent wall candidate information; and calculates the degree of reliability that the intermittent wall candidate information is detection information of the intermittent wall, on the basis of the intermittent wall candidate information and information of the three-dimensional object detected by the three-dimensional object detection processing unit.

Description

Environmental recognition device and recording medium

[0001] The present disclosure relates to an environmental recognition device and a recording medium.

[0002] In recent years' advanced driving assistance systems for vehicles, environmental recognition technologies that utilize imaging devices such as stereo cameras have been used. For example, in Patent Document 1, a technology for detecting sidewalls of a road such as walls, guardrails, and plantings has been proposed. Specifically, Patent Document 1 discloses a technology for generating two-dimensional distribution information of an object that associates at least the horizontal distance and the depth distance of the object, detecting a continuous region that is continuous in the depth direction in the two-dimensional distribution information, and determining whether the continuous region indicates a detection target object, thereby easily detecting a continuous three-dimensional object.

[0003] International Publication No. 2017 / 115732

[0004] However, the technology disclosed in Patent Document 1 targets three-dimensional objects having a length greater than or equal to a certain length. There are also walls that do not meet a certain length, such as short walls or intermittently existing walls (hereinafter, these walls are collectively referred to as "intermittent walls") on the sidewalls existing on ordinary roads. Therefore, with the technology disclosed in Patent Document 1, there is a possibility that the scenes where the information of the sidewall detection result can be used for driving assistance are limited.

[0005] The present disclosure has been made in view of the above problems, and aims to make the information of the detection result of an intermittent wall available for driving assistance.

[0006] To solve the above problems, according to one aspect of this disclosure, the present invention comprises at least a three-dimensional object detection processing unit that detects three-dimensional objects based on images captured by a stereo camera, and a discontinuous wall detection processing unit that detects discontinuous walls based on the captured images, wherein the discontinuous wall detection processing unit obtains a plurality of detection points by projecting unit blocks consisting of a predetermined number of pixels that serve as processing units in the captured images, where the height from the road surface is within a predetermined range, onto a two-dimensional plane consisting of the depth direction and the width direction as viewed from the stereo camera, and groups the detection points that are within a predetermined distance from each other, and obtains a quadratic approximation model of the grouped detection point group. An environmental recognition device is provided that divides the above two-dimensional plane into first distance ranges in the depth direction, and for each of the divisions in the horizontal direction that include the above quadratic approximation model in each of the above first distance ranges, determines candidate wall lines for each section, determines candidate wall information including the detected points and candidate wall lines included in the section, calculates the reliability of the candidate wall information for each section based on the detected points and candidate wall lines included in the section and the information of the three-dimensional object detected by the three-dimensional object detection processing unit, and generates output information based on the candidate wall information and the reliability information.

[0007] Furthermore, in order to solve the above problem, according to another aspect of this disclosure, one or more processors are made to perform the following: detecting three-dimensional objects based on images captured by a stereo camera and detecting intermittent walls based on the captured images, and in detecting the intermittent walls, multiple detection points are obtained by projecting unit blocks consisting of a predetermined number of pixels, which are processing units in the captured images, and unit blocks whose height from the road surface is within a predetermined range onto a two-dimensional plane consisting of the depth direction and the width direction as viewed from the stereo camera, and the detection points that are within a predetermined distance from each other are grouped together, and a quadratic approximation model of the grouped detection point group is obtained. A non-temporary tangible recording medium is provided which records a program that performs the following actions: divides the two-dimensional plane in the depth direction into first distance ranges, divides the horizontal range including the quadratic approximation model in each of the first distance ranges, determines candidate wall lines for each of the divisions, determines candidate wall information including the detected points and candidate wall lines included in the division, calculates the reliability of the candidate wall information for each division based on the detected points and candidate wall lines included in the division and the information of the detected three-dimensional object, and generates output information based on the candidate wall information and the reliability information.

[0008] As explained above, this disclosure makes it possible to use the information obtained from the detection of intermittent walls for driver assistance.

[0009] This is a schematic diagram showing an example configuration of a vehicle equipped with an environmental recognition device according to the present disclosure. This is a block diagram showing an example configuration of the environmental recognition device according to the same embodiment. This is a flowchart of the intermittent wall detection process performed by the environmental recognition device according to the same embodiment. This is an explanatory diagram showing the process of obtaining a plurality of detection points projected onto the xz two-dimensional plane by the environmental recognition device according to the same embodiment. This is an explanatory diagram showing the process of extracting effective segments by the environmental recognition device according to the same embodiment. This is an explanatory diagram showing the process of grouping effective segments and detection points by the environmental recognition device according to the same embodiment. This is a flowchart of the process of obtaining a quadratic approximation model and wall candidate information by the environmental recognition device according to the same embodiment. This is an explanatory diagram showing a fixed pattern when using RANSAC with the environmental recognition device according to the same embodiment. This is an explanatory diagram showing the process of obtaining a quadratic approximation model and wall candidate information by the environmental recognition device according to the same embodiment. This is an explanatory diagram showing the process of obtaining a quadratic approximation model and wall candidate information by the environmental recognition device according to the same embodiment. This is an explanatory diagram showing the process of obtaining wall candidate information for each section by the environmental recognition device according to the same embodiment. This is an explanatory diagram showing the process of searching for the starting point of a wall candidate by the environmental recognition device according to the same embodiment. This is an explanatory diagram showing the process of calculating the reliability by the environmental recognition device according to the same embodiment. This is an explanatory diagram showing the process of integrating wall candidates by the environmental recognition device according to the same embodiment. This is an explanatory diagram showing the process by which the environmental recognition device according to the same embodiment excludes wall candidates with different angles as noise. This is an explanatory diagram showing the process by which the environmental recognition device according to the same embodiment generates interpolation line data. This is an explanatory diagram showing the process by which the environmental recognition device according to the same embodiment generates detection information for intermittent walls. This shows a flowchart of the interpolation process using past intermittent wall information by the environmental recognition device according to the same embodiment. This is an explanatory diagram showing the process by which the environmental recognition device according to the same embodiment updates the position of representative points of past intermittent wall information based on the amount of vehicle movement. This is an explanatory diagram showing the process by which the environmental recognition device according to the same embodiment updates the information for determination. This is an explanatory diagram showing the process by which the environmental recognition device according to the same embodiment compares the detection information of intermittent walls detected in the current calculation cycle with past intermittent wall information.This is an explanatory diagram showing the process by which the environmental recognition device according to the same embodiment determines whether or not there is past intermittent wall information that can be interpolated outside the search range of the current xz two-dimensional plane. This is an explanatory diagram showing the process by which the environmental recognition device according to the same embodiment determines whether or not there is past intermittent wall information that can be interpolated within the search range of the current xz two-dimensional plane. This is an explanatory diagram showing the process by which the environmental recognition device according to the same embodiment reflects past intermittent wall information in the detection information of the current intermittent wall. This is an explanatory diagram showing the process by which the environmental recognition device according to the same embodiment reflects past intermittent wall information in the detection information of the current intermittent wall.

[0010] Preferred embodiments of this disclosure will be described in detail below with reference to the attached drawings. The specific dimensions, materials, numerical values, etc., shown in the following embodiments are merely examples to facilitate understanding of the invention and do not limit the present invention unless otherwise specified. In this specification and drawings, components having substantially the same functional configuration are denoted by the same reference numerals to avoid redundant explanations.

[0011] In this specification, "intermittent wall" includes not only walls that exist intermittently but also walls of short length. Furthermore, "side wall" refers to a continuous wall having a length of a predetermined or greater length.

[0012] <1. Overall Vehicle Configuration> First, an example of the overall configuration of a vehicle equipped with the environmental recognition device according to the embodiment of this disclosure will be described.

[0013] Figure 1 is a schematic diagram showing an example of the configuration of a vehicle 1 equipped with an environmental recognition device 50 according to this embodiment. The vehicle 1 shown in Figure 1 is a four-wheel drive vehicle that transmits the drive torque output from the drive source 9 to the left front wheel 3FL, the right front wheel 3FR, the left rear wheel 3RL, and the right rear wheel 3RR (hereinafter collectively referred to as "wheels 3" unless otherwise specified). The drive source 9 may be an internal combustion engine such as a gasoline engine or a diesel engine, a drive motor, or both.

[0014] Vehicle 1 may be an electric vehicle equipped with two drive motors, for example, a front-wheel drive motor and a rear-wheel drive motor, or an electric vehicle equipped with a drive motor corresponding to each wheel 3. Vehicle 1 may also be a two-wheel drive vehicle with front-wheel drive or rear-wheel drive. Furthermore, if vehicle 1 is an electric vehicle or a hybrid electric vehicle, vehicle 1 may be equipped with a secondary battery that stores the power supplied to the drive motors, or a generator such as a motor or fuel cell that generates the power charged to the battery.

[0015] Vehicle 1 is equipped with a drive source 9, an electric steering device 15, and brake devices 17FL, 17FR, 17RL, and 17RR (hereinafter collectively referred to as "brake device 17" unless otherwise specified) as equipment used for controlling the operation of Vehicle 1. The drive source 9 outputs drive torque that is transmitted to the front wheel drive shaft 5F and the rear wheel drive shaft 5R via a transmission (not shown), a front wheel differential mechanism 7F, and a rear wheel differential mechanism 7R. The drive of the drive source 9 and the transmission is controlled by a control device 40 which is configured to include one or more electronic control units (ECUs).

[0016] The electric steering system 15 includes an electric motor and gear mechanism (not shown) and is controlled by the control device 40 to adjust the steering angles of the left front wheel 3FL and the right front wheel 3FR. The control device 40 controls the electric steering system 15 so that the vehicle 1 travels in a predetermined position when the driver assistance function switch is turned on and the driving mode is set to the driver assistance mode. The control device 40 also controls the electric steering system 15 based on the steering angle of the steering wheel 13 by the driver when the driving mode is set to the manual driving mode.

[0017] The braking system 17 applies braking force to the front, rear, left, and right wheels 3, respectively. The braking system 17 is, for example, a hydraulic braking system. A hydraulic control unit 41 controls the hydraulic pressure supplied to each braking system 17, thereby generating a predetermined braking force. If the vehicle 1 is an electric vehicle or a hybrid electric vehicle, the braking system 17 is used in conjunction with regenerative braking by the drive motor.

[0018] The control device 40 includes one or more electronic control devices that control the drive of the power source 9, the electric steering device 15, and the hydraulic control unit 41. The control device 40 may also have a function to control the drive of the transmission that changes the speed of the output from the power source 9 and transmits it to the wheels 3. The control device 40 is configured to perform driving assistance processing for the vehicle 1 using information on obstacles recognized by the environment recognition device 50. For example, the control device 40 performs lane keeping control, which controls the electric steering device 15 so that the vehicle 1 does not deviate from the boundary line of the driving lane. However, the driving assistance processing is not limited to lane keeping control, and may be driving assistance processing by conventionally known advanced driving assistance functions.

[0019] Vehicle 1 is also equipped with a pair of left and right stereo cameras 31FL and 31FR, a vehicle status sensor 35, a GNSS (Global Navigation Satellite System) antenna 37, a switch 39, a notification device 43, and an environmental recognition device 50. The vehicle status sensor 35, GNSS antenna 37, switch 39, notification device 43, and environmental recognition device 50 are connected to the control device 40 via a dedicated line or communication means such as CAN (Controller Area Network) or LIN (Local Internet).

[0020] The stereo cameras 31FL and 31FR capture images of the area in front of the vehicle 1 and generate a pair of stereo images (captured images). The stereo cameras 31FL and 31FR are imaging devices equipped with image sensors such as CCD (Charged-Coupled Devices) or CMOS (Complementary Metal-Oxide-Semiconductor). The stereo cameras 31FL and FR are connected to the environment recognition device 50 via wired or wireless communication means and transmit the generated stereo images to the environment recognition device 50.

[0021] In addition to the stereo cameras 31FL and 31FR, the vehicle 1 may also be equipped with a camera that photographs the area behind the vehicle 1, or a camera mounted on a side mirror or the like that photographs the left rear or right rear. Furthermore, in addition to the stereo cameras 31FL and 31FR, the vehicle 1 may also be equipped with a radar sensor such as a LiDAR or millimeter-wave radar, or another distance measuring sensor such as an ultrasonic sensor.

[0022] The vehicle state sensor 35 is one or more sensors that detect the operating state and behavior of the vehicle 1. The vehicle state sensor 35 includes, for example, a steering angle sensor, an accelerator position sensor, a brake stroke sensor, and a brake pressure sensor. It also includes, for example, a vehicle speed sensor, an acceleration sensor, and an angular velocity sensor. The vehicle state sensor 35 transmits sensor signals indicating the detected information to the control device 40.

[0023] The GNSS antenna 37 receives satellite signals transmitted from satellites such as GPS (Global Positioning System). The GNSS antenna 37 transmits the vehicle 1's position information, which is included in the received satellite signals, to the control device 40.

[0024] Switch 39 is operated by the driver to switch on or off the driver assistance function, which is a system that takes over the driving task of the vehicle 1. Switch 39 may be a physical switch, a touch panel, or a voice input device.

[0025] The environmental recognition device 50 comprises one or more processors such as CPUs (Central Processing Units) and one or more memories connected to the one or more processors in a communicative manner. The environmental recognition device 50 functions as a device that performs processing to recognize the environment around the vehicle 1 by having one or more processors execute a computer program. The computer program is a computer program that causes the processor to execute the operations to be performed by the environmental recognition device 50, which will be described later. The computer program executed by the processor may be recorded on a recording medium that functions as a memory unit provided in the environmental recognition device 50, or it may be recorded on a recording medium built into the environmental recognition device 50 or on any external recording medium that can be attached to the environmental recognition device 50.

[0026] Recording media for storing computer programs may include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs, DVDs, and Blu-ray®; magneto-optical media such as floppy disks; memory elements such as RAM (Random Access Memory) or ROM (Read Only Memory); flash memory such as USB memory and SSDs; and other media capable of storing programs.

[0027] The environmental recognition device 50 performs a process to recognize the environment around the vehicle 1 based on the data transmitted from the stereo cameras 31FL and 31FR. The environmental recognition device 50 performs a process to detect three-dimensional objects based on the images captured by the stereo cameras 31FL and 31FR, and a process to detect intermittent walls based on the images captured by the stereo cameras 31FL and 31FR. The environmental recognition device 50 outputs the recognized information about the surrounding environment to the control device 40.

[0028] The notification device 43 is driven by the control device 40 and notifies the occupants of various information by means of image display, audio output, etc. The notification device 43 includes, for example, a display device provided in the instrument panel and a speaker provided in the vehicle 1. The display device may be a display device of a navigation system.

[0029] <2. Environmental Recognition Device> Next, the environmental recognition device 50 according to this embodiment will be described in detail.

[0030] (2-1. Configuration Example) Figure 2 is a block diagram showing the configuration of the part of the environmental recognition device 50 according to this embodiment that is related to the process of detecting intermittent walls. The environmental recognition device 50 is connected to stereo cameras 31FL, 31FR (hereinafter referred to as stereo camera 31) via a dedicated line or a communication means such as CAN or LIN. The environmental recognition device 50 is also connected to a control device 40 via a dedicated line or a communication means such as CAN or LIN.

[0031] The environmental recognition device 50 comprises a processing unit 51 and a storage unit 53. The processing unit 51 is configured with one or more processors such as CPUs. Part or all of the processing unit 51 may be configured with updatable components such as firmware, or it may be a program module executed by commands from the CPU or the like.

[0032] The storage unit 53 is composed of one or more memory elements (RAM and ROM) that are connected to the processing unit 51 in a communicative manner. However, the number and type of storage units 53 are not particularly limited. The storage unit 53 stores computer programs executed by the processing unit 51, various parameters used in arithmetic processing, detection data, calculation results, and other data. A portion of the storage unit 53 is used as the work area of ​​the processing unit 51.

[0033] In addition, the environmental recognition device 50 is equipped with an interface (not shown) for transmitting and receiving data with the stereo cameras 31FL, 31FR and the control device 40.

[0034] The processing unit 51 includes a position derivation processing unit 61, a three-dimensional object detection processing unit 63, a discontinuous wall detection processing unit 65, a side wall detection processing unit 67, and an output information generation processing unit 69. The position derivation processing unit 61, the three-dimensional object detection processing unit 63, the discontinuous wall detection processing unit 65, the side wall detection processing unit 67, and the output information generation processing unit 69 are functions realized by the execution of a program by one or more processors. Note that some or all of the position derivation processing unit 61, the three-dimensional object detection processing unit 63, the discontinuous wall detection processing unit 65, the side wall detection processing unit 67, and the output information generation processing unit 69 may be configured by hardware such as analog circuits.

[0035] The functions of the position derivation processing unit 61, the three-dimensional object detection processing unit 63, the intermittent wall detection processing unit 65, the side wall detection processing unit 67, and the output information generation processing unit 69 will be briefly described below, followed by a detailed explanation of the specific processing operations.

[0036] (Position Derivation Processing Unit) The position derivation processing unit 61 performs a process to derive the three-dimensional position in real space for each unit block consisting of a predetermined number of pixels, which serves as the processing unit, from a pair of stereo images output from the stereo camera 31. A unit block is represented, for example, by an array of 4 horizontal pixels × 4 vertical pixels, but it may be set to any number of pixels. The position derivation processing unit 61 generates a distance image that associates the derived three-dimensional position with each unit block.

[0037] (3D Object Detection Processing Unit) The 3D object detection processing unit 63 performs a process to detect 3D objects present around the vehicle 1 based on the 3D position information for each unit block generated by the position derivation processing unit 61. For example, the 3D object detection processing unit 63 identifies 3D objects by grouping blocks that are located above the road surface, have the same color value, and whose 3D position difference in the distance image is within a predetermined range. Specifically, the 3D object detection processing unit 63 groups blocks whose x-coordinate difference, y-coordinate difference, and z-coordinate difference in the distance image are within a predetermined range (for example, 0.1 m) assuming that they correspond to the same 3D object.

[0038] (Intermittent Wall Detection Processing Unit) The intermittent wall detection processing unit 65 executes a process to generate intermittent wall information. The intermittent wall detection processing unit 65 executes a process to detect intermittent walls based on the three-dimensional position information for each unit block generated by the position derivation processing unit 61. In this embodiment, the intermittent wall detection processing unit 65 obtains a plurality of detection points by projecting unit blocks in the captured image whose height from the road surface is within a predetermined range onto the xz two-dimensional plane. The intermittent wall detection processing unit 65 also groups the detection points that are within a predetermined distance and obtains a quadratic approximation model of the grouped detection point group. Furthermore, the intermittent wall detection processing unit 65 divides the two-dimensional plane into first distance ranges in the depth direction, and divides the horizontal range in each first distance range so as to include the quadratic approximation model. For each of these divisions, it obtains candidate wall lines and intermittent wall information that includes the detection points and candidate wall lines contained in the division. Furthermore, the intermittent wall detection processing unit 65 calculates the reliability of the wall candidate information for each section based on the detection points and wall candidate lines included in the section, and the information of the three-dimensional objects detected by the three-dimensional object detection processing unit 63. Then, the intermittent wall detection processing unit 65 generates intermittent wall information based on the wall candidate information and the reliability information.

[0039] (Sidewall Detection Processing Unit) The sidewall detection processing unit 67 performs a process to generate sidewall information. In this embodiment, the sidewall detection processing unit 67 obtains a plurality of detection points by projecting unit blocks with a predetermined height range from the road surface onto a two-dimensional plane consisting of the depth direction (z-axis) and the lateral direction (x-axis) as seen from the stereo camera 31, based on the three-dimensional position information for each unit block generated by the position derivation processing unit 61, and identifies a group of detection points that are estimated to be sidewalls based on the plurality of detection points, and generates sidewall information. In this embodiment, the sidewall detection processing unit 67 only needs to be able to identify a group of detection points that are estimated to be continuous sidewalls having a length of a predetermined or greater from among the plurality of detection points projected onto the xz two-dimensional plane, and the specific method is not particularly limited. The sidewall detection processing unit 67 also performs a process to calculate the reliability of the sidewall information.

[0040] (Output Information Generation Processing Unit) The output information generation processing unit 69 generates output information to be output to the control device 40 based on the side wall information and the intermittent wall information, and outputs the generated output information to the control device 40. The control device 40 executes a process of assisting the driving of the vehicle 1 using the acquired output information.

[0041] (2-2. Operation) Next, referring to the drawings as appropriate, the processing operation of the environment recognition device 50 according to the present embodiment will be described in detail.

[0042] FIG. 3 shows a flowchart of the intermittent wall detection process executed by the environment recognition device 50. When the system of the vehicle 1 is started (step S11), the processing unit 51 updates the intermittent wall information calculated in the past (step S13). For example, the processing unit 51 updates the intermittent wall information calculated in the most recent predetermined number of calculation cycles as the past intermittent wall information, as will be described later. The most recent predetermined number of calculation cycles may be set to any appropriate number, but for example, it can be set to the most recent 5 calculation cycles.

[0043] Next, the position derivation processing unit 61 of the processing unit 51 acquires a pair of stereo images from the stereo camera 31 and executes a position derivation process (step S15). For example, the position derivation processing unit 61 uses a pattern matching technique to derive parallax information including the parallax and the image position indicating the position within the image of an arbitrary unit block. Then, the position derivation processing unit 61 searches for a block corresponding to a unit block arbitrarily extracted from one of the pair of stereo images in the other image. The unit block may be, for example, a region of 4 pixels horizontally × 4 pixels vertically, but the size of the unit block is not limited to the above example. The position derivation processing unit 61 performs the process of deriving parallax information for each unit block for all the unit blocks in the detection region set in the image. The position derivation processing unit 61 generates a distance image in which the derived parallax information is associated with each unit block.

[0044] The position derivation processing unit 61 converts the derived parallax information for each unit block into a position (three-dimensional position) on a three-dimensional coordinate system with the horizontal direction (vehicle width direction) in the real space as the x-axis, the height direction as the y-axis, and the depth direction as the z-axis, according to the principle of triangulation. At this time, the position derivation processing unit 61 derives the height position (y coordinate) of each unit block from the road surface based on the z coordinate of each unit block and the image distance between the point on the road surface at the same z coordinate as the unit block and the unit block. Then, the position derivation processing unit 61 stores the derived three-dimensional position in association with the distance image again.

[0045] Note that the process of deriving parallax information and the process of calculating the three-dimensional position by the position derivation processing unit 61 are merely examples, and may be performed using various known techniques.

[0046] Next, the intermittent wall detection processing unit 65 obtains a plurality of detection points by projecting unit blocks within a predetermined range of height from the road surface among the unit blocks in the distance image onto a two-dimensional plane composed of the horizontal direction (x direction) and the depth direction (z direction) as viewed from the stereo camera 31 (step S17).

[0047] FIG. 4 shows an example of a process of obtaining a plurality of detection points by projecting unit blocks within a predetermined range of height from the road surface onto the xz two-dimensional plane. FIG. 4 is an example of generating a distance histogram by mapping detection points obtained by projecting unit blocks with a height exceeding zero onto the xz two-dimensional plane divided into unit segments with a horizontal length Lx1 of 492 mm and a depth length Lz of 8,192 mm. The unit segment refers to an area obtained by dividing the xz two-dimensional plane into first distance ranges (Lz) in the depth direction and second distance ranges (Lx1) in the horizontal direction. In the illustrated example, the detection range is the xyz three-dimensional space on the xz two-dimensional plane with 127 segments in the horizontal direction and 44 segments in the depth direction, that is, the xz two-dimensional plane of 62,484 mm × 360,448 mm.

[0048] Note that the unit blocks projected onto the two-dimensional plane are all unit blocks with a height exceeding zero existing at a position higher than the road surface, but at least one of the upper limit and the lower limit may be set to any appropriate value.

[0049] Next, the intermittent wall detection processing unit 65 groups together the detection points that are within a predetermined distance from each other according to a given criterion (step S19).

[0050] Figures 5 and 6 show an example of the process of grouping detected points. As shown in Figure 5, the intermittent wall detection processing unit 65 extracts segments containing at least one detected point (hereinafter also referred to as "effective segments") from the distance histogram on which the detected points are mapped. For example, the intermittent wall detection processing unit 65 sequentially searches for segments along the lateral direction in the distance histogram and extracts effective segments at change points where segments that do not contain a detected point change to effective segments containing a detected point. In other words, the intermittent wall detection processing unit 65 extracts effective segments on the road surface where a difference in elevation appears in the lateral direction as viewed from the vehicle 1. The intermittent wall detection processing unit 65 repeats the process of extracting effective segments where the above-mentioned difference in elevation appears from the near side to the far side as viewed from the vehicle 1.

[0051] Furthermore, when the intermittent wall detection processing unit 65 searches for segments along the lateral direction, it may search from the left end to the right end, or from the right end to the left end, or it may search along the lateral direction from the front of the vehicle 1 (position x=0 in Figure 5) toward both the left and right sides, or it may search from the left end and the right end toward the front of the vehicle 1. In addition, the intermittent wall detection processing unit 65 may repeat the process of extracting effective segments where the above-mentioned height difference appears, from the far side toward the near side as viewed from the vehicle 1, and the order is not particularly limited.

[0052] Next, as shown in Figure 6, the intermittent wall detection processing unit 65 groups together multiple consecutive effective segments from among the extracted effective segments. For example, the intermittent wall detection processing unit 65 groups together effective segments that are consecutive in the horizontal or depth direction from among the extracted effective segments. The effective segments to be grouped may also include effective segments that are consecutive in the diagonal direction. Then, the intermittent wall detection processing unit 65 groups together the detection points included in the grouped effective segments.

[0053] Next, the intermittent wall detection processing unit 65 obtains a quadratic approximation model of the detected point grouped in step S19, determines candidate wall lines for each section obtained by dividing the xz two-dimensional plane according to predetermined conditions, and obtains candidate wall information including the detected points and candidate wall lines included in the section (step S21).

[0054] Here, the xz two-dimensional plane is divided into the left side region of vehicle 1, the front region of vehicle 1, and the right side region of vehicle 1, depending on the lateral position. When detecting intermittent walls, taking into account that there is a limit to the distance vehicle 1 can move in one calculation cycle, the intermittent wall detection processing unit 65 excludes the detection point group located in the front region of vehicle 1 and calculates a quadratic approximation model of the detection point group located in the left side region and the right side region of vehicle 1. In the following explanation, the entire range of the xz two-dimensional plane is referred to as the "search range," and the regions on both the left and right sides of vehicle 1, excluding the front region of vehicle 1, are also referred to as the "effective range."

[0055] Figure 7 shows a flowchart of the process for obtaining a quadratic approximation model of the grouped detection point cloud and obtaining wall candidate information for each predetermined section. The intermittent wall detection processing unit 65 performs the following processing for each grouped detection point cloud. The processing described below calculates a quadratic approximation model using the so-called RANSAC method, but in this embodiment, it includes processing to reduce the variability of the calculation results.

[0056] First, the intermittent wall detection processing unit 65 selects multiple detection points from the grouped detection point set and calculates a candidate model consisting of a quadratic approximation curve, for example, using the least squares method (step S41). Next, the intermittent wall detection processing unit 65 counts the number of detection points Np from the grouped detection point set that fit the calculated candidate model (step S43). A fitting detection point is identified, for example, as a detection point whose distance from the candidate model is less than or equal to a predetermined threshold.

[0057] Next, the intermittent wall detection processing unit 65 determines whether the number of detection points Np that fit the calculated candidate model exceeds a predetermined threshold Np0 (step S45). The predetermined threshold Np0 is set to an arbitrary appropriate value, taking into consideration the acceptable range of accuracy of the final calculated quadratic approximation model. If the intermittent wall detection processing unit 65 does not determine that the number of detection points Np that fit the calculated candidate model exceeds the predetermined threshold Np0 (S45 / No), the process returns to step S41.

[0058] On the other hand, if the discontinuous wall detection processing unit 65 determines that the number of detection points Np that fit the calculated candidate model exceeds a predetermined threshold Np0 (S45 / Yes), it saves the calculated candidate model as an evaluation model (step S47). Next, the discontinuous wall detection processing unit 65 determines whether the number of saved evaluation models Nm is greater than or equal to a predetermined threshold Nm0 (step S49). If the discontinuous wall detection processing unit 65 does not determine that the number of saved evaluation models Nm is greater than or equal to a predetermined threshold Nm0 (S49 / No), it returns to step S41 and repeats the generation of evaluation models until the number of evaluation models Nm reaches the predetermined threshold Nm0.

[0059] In RANSAC, which calculates the parameters of a mathematical model by excluding the influence of outliers from data containing outliers, if detection points are selected randomly to create an evaluation model, the calculation results may change even with the same input data. This variability in calculation results may reduce the performance equivalence of the driving assistance function using intermittent wall information. In this embodiment, in order to reduce the effect of this randomness, the intermittent wall detection processing unit 65 selects a predetermined number of detection points according to a predetermined fixed pattern based on the total number of grouped detection points, and generates a predetermined number of evaluation models.

[0060] Figure 8 shows an example of a fixed pattern when using RANSAC. In the illustrated example, when the total number of detection points Na is Na1 to Na2, the number of detection points to select Nb is set to Nb1 and the number of evaluation models to generate Nm0 is set to Nm0_1; when the total number of detection points Na is Na3 to Na4, the number of detection points to select Nb is set to Nb2 and the number of evaluation models to generate Nm0 is set to Nm0_2; when the total number of detection points Na is Na5 to Na6, the number of detection points to select Nb is set to Nb3 and the number of evaluation models to generate Nm0 is set to Nm0_3; and when the total number of detection points Na is Na7 or more, the number of detection points to select Nb is set to Nb4 and the number of evaluation models to generate Nm0 is set to Nm0_4. The intermittent wall detection processing unit 65 selects detection points according to the number of detection points to select Nb set according to the total number of grouped detection points Na, and calculates candidate models of quadratic approximation curves using the least squares method. The discontinuous wall detection processing unit 65 then selects a candidate model as an evaluation model if the number of candidate points Np that fit the candidate model exceeds a predetermined threshold Np0 from the grouped detection point set. The discontinuous wall detection processing unit 65 repeats the generation of candidate models (evaluation models) by changing the selected detection points until the number of generated evaluation models Nm reaches the number of evaluation models Nm0, which is set according to the total number of grouped detection points Na.

[0061] Further criteria may be established as fixed patterns when generating the evaluation model. For example, additional criteria may be applied to prevent bias in the selection of detection points, such as fixing one or more detection points to be selected based on the ratio of the total number of grouped detection points to the number of detection points to be selected.

[0062] In step S49 described above, if the discontinuous wall detection processing unit 65 determines that the number Nm of saved evaluation models is greater than or equal to a predetermined threshold Nm0 (S49 / Yes), it excludes detection points that do not fit any of the generated evaluation models as outliers (step S51). Next, the discontinuous wall detection processing unit 65 calculates a quadratic approximation model from the remaining group of detection points from which the outliers have been excluded (step S53).

[0063] Figures 9 and 10 show an example of a quadratic approximation model calculated from the detected point cloud. Note that the aspect ratio of the unit segments shown in Figures 9 and 10 is different from the aspect ratio of the segments shown in Figures 5 to 7 above. In the example shown in Figure 9, the intermittent wall detection processing unit 65 excludes detection points projected onto the xz two-dimensional plane that are included in an effective segment different from the effective segment of the target group for which the quadratic approximation model is calculated. The intermittent wall detection processing unit 65 selects five detection points from the detection points included in the effective segment of the target group according to a fixed pattern to generate 10 evaluation models. Furthermore, the intermittent wall detection processing unit 65 excludes detection points that do not fit any of the 10 evaluation models as outliers and calculates a quadratic approximation model using the remaining detection points. The intermittent wall detection processing unit 65 calculates a quadratic approximation model for each grouped set of detection points.

[0064] Next, the intermittent wall detection processing unit 65 obtains wall candidate information for each section of the xz two-dimensional plane divided according to predetermined conditions (step S55). The intermittent wall detection processing unit 65 records the calculated wall candidate information in association with each section.

[0065] Figure 11 is a diagram illustrating the divisions based on predetermined conditions and the wall candidate information obtained for each division. Figure 11 shows a portion of the quadratic approximation model and detection point cloud shown in Figure 10. The intermittent wall detection processing unit 65 divides the xz two-dimensional plane into first distance ranges Lz in the depth direction, and in each first distance range Lz, it includes the quadratic approximation model in the lateral range Lx2 １ , Lx2 ２ , Lx2 ３ A section is defined. The depth of the section (Lz) is the same as the depth of the unit segment used when calculating the quadratic approximation model. On the other hand, since each section is divided to include the quadratic approximation model in its respective first distance range Lz in the depth direction, the lateral lengths of each section are different (Lx2). １ ≠Lx2 ２ ≠Lx2 ３ ).

[0066] The intermittent wall detection processing unit 65 determines, for each section, whether or not a detection point exists, the x-coordinates of the leftmost and rightmost detection points if a detection point exists, and the average value of the x-coordinates of the detection points if a detection point exists. The intermittent wall detection processing unit 65 also connects diagonals along the quadratic approximation model contained in each section, creating candidate wall lines for each section. In the example shown in Figure 11, the quadratic approximation model draws a quadratic approximation curve, while the candidate wall lines for each section draw straight lines. The intermittent wall detection processing unit 65 records the information on whether or not a detection point exists in each section, the information on the x-coordinates of the leftmost and rightmost detection points if a detection point exists, the information on the average value of the x-coordinates of the detection points if a detection point exists, and the information on the candidate wall lines as a set of candidate wall information, associated with each section. The intermittent wall detection processing unit 65 sets sections for all the calculated quadratic approximation models and obtains candidate wall information.

[0067] Returning to the flowchart in Figure 3, the intermittent wall detection processing unit 65 calculates a quadratic approximation model and performs the process of obtaining wall candidate information, and then searches for the starting point of the wall candidate (step S22). For example, based on the wall candidate information, if there are consecutive wall candidate lines for multiple sections among the wall candidate lines that exist in the effective range, the intermittent wall detection processing unit 65 considers the consecutive wall candidate lines as a single wall candidate. In other words, a wall candidate refers to a combination of multiple connected wall candidate lines. If there are multiple wall candidates, the intermittent wall detection processing unit 65 selects the wall candidate located at the closest distance from the stereo camera 31 (vehicle 1) as the starting point of the wall candidate (first wall candidate).

[0068] Figure 12 is an explanatory diagram illustrating the overview of the process for searching for the starting point of a wall candidate. In the illustrated example, the intermittent wall detection processing unit 65 extracts wall candidates from the effective range, which is the area on both the left and right sides of the vehicle 1 in the xz two-dimensional plane shown in Figure 6, where the length L in the depth direction is L1 or more and the width W in the lateral direction is within W1, and the entire wall candidate is located within Lz_α in the depth direction from the position of the stereo camera 31. If there are multiple matching wall candidates, the intermittent wall detection processing unit 65 sets the wall candidate closest to the position of the stereo camera 31 (vehicle 1) as the starting point of the wall candidate (first wall candidate). Since the intermittent wall information is mainly used to avoid collisions between the vehicle 1 and intermittent walls and side walls, the intermittent wall detection processing unit 65 sets the wall candidate closest to the vehicle 1 as the starting point of the wall candidate.

[0069] The condition for the length L1 in the depth direction indicates the minimum depth of the intermittent wall to be detected and is set to prevent the false detection of objects other than intermittent walls as intermittent walls. The condition for the width W1 in the lateral direction is set so that when the detection target is an intermittent wall, the lateral width W as seen from the vehicle 1 falls within a predetermined range, and is set to prevent the false detection of objects other than intermittent walls as intermittent walls. The condition for the range in which the entire wall candidate exists is set so that the detection target of the intermittent wall detection processing unit 65 in this embodiment is a short side wall or an intermittently existing side wall, and walls that are continuous for a predetermined length or longer are excluded from the target. Note that the above conditions are just examples and may be set to any appropriate value.

[0070] Next, the intermittent wall detection processing unit 65 calculates the reliability of the wall candidate information for each section (step S23). In this embodiment, the intermittent wall detection processing unit 65 calculates the reliability of the wall candidate information for each section based on the detection points and wall candidate lines included in each section and the information of the three-dimensional object detected by the three-dimensional object detection processing unit 63. For example, for each section from which wall candidate information has been obtained, the intermittent wall detection processing unit 65 calculates the reliability of the wall candidate information based on the number of detection points included in the section and the distance between the wall candidate line and the three-dimensional object projection line obtained by projecting the outline of the three-dimensional object detected by the three-dimensional object detection processing unit 63 onto the xz two-dimensional plane.

[0071] Figure 13 is an explanatory diagram showing an example of the process for calculating confidence. Note that the sections and candidate wall lines shown to explain the calculation methods for the first confidence value C1 and the second confidence value C2 are for illustrating the concepts used to calculate the first confidence value C1 and the second confidence value C2, respectively, and do not represent the same data.

[0072] The intermittent wall detection processing unit 65 determines a first accuracy value C1 for each section according to the number of detection points present within the section. In the example shown in Figure 13, the first accuracy value C1 is a value within the range of 0 to 100. For example, the first accuracy value C1 is 0 when the number of detection points is zero, 30 when the number of detection points is 1 to 4, 80 when the number of detection points is 5 to 7, and 100 when the number of detection points is 8 or more. The more detection points there are in a section, the higher the probability that they are detection points of three-dimensional objects with a height greater than zero from the road surface, and the first accuracy value C1 approaches 100.

[0073] Furthermore, the intermittent wall detection processing unit 65 acquires three-dimensional object information, indicated by three-dimensional coordinates, from the three-dimensional object detection processing unit 63, and generates a three-dimensional object projection line by projecting the outline of the said three-dimensional object information onto the xz two-dimensional plane. The three-dimensional object information obtained from the stereo image is indicated by the surface of the three-dimensional object visible from the stereo camera 31, and the three-dimensional object projection line, which is the projection of said three-dimensional object information onto the xz two-dimensional plane, represents the outline of the three-dimensional object. Therefore, the smaller the lateral deviation between the wall candidate line and the three-dimensional object projection line, the higher the probability that the detection point is for a short three-dimensional object. For example, a maximum value for the lateral deviation is set in advance, and the intermittent wall detection processing unit 65 sets the second accuracy value C2 to 0 when the lateral distance between the average value of the x-coordinate of the detection point in each section and the three-dimensional object projection line is greater than or equal to the maximum value, and sets the second accuracy value C2 to 100 when the distance is zero, and calculates the second accuracy value C2 in 10 steps according to each distance obtained by dividing the maximum value into 10 equal parts.

[0074] The intermittent wall detection processing unit 65 then calculates the reliability of the wall candidate information for each section using the following formula: The coefficient α is a weighting coefficient and may be set within the range of 0 to 1 depending on the importance of the first accuracy value C1 and the second accuracy value C2. Reliability = α × C1 + (1 - α) × C2 Therefore, the calculated reliability will be a value within the range of 0 to 100. The larger the reliability value, the higher the probability that each wall candidate information indicates an intermittent wall.

[0075] The intermittent wall detection processing unit 65 records the reliability information of the wall candidate information for each section, which has been calculated, as intermittent wall information, associating it with each section.

[0076] Next, if there are multiple wall candidates, the intermittent wall detection processing unit 65 integrates the multiple wall candidates (step S25). By integrating multiple wall candidates that satisfy predetermined conditions, the intermittent wall detection processing unit 65 makes the intermittent wall available for use in the driving support processing as information for a single side wall. If there is no starting point for the wall candidate searched in step S22, the intermittent wall detection processing unit 65 skips the process of integrating multiple wall candidates. In this case, the following steps S27 to S29 are also skipped.

[0077] Figure 14 is an explanatory diagram illustrating the process of integrating multiple wall candidates. The intermittent wall detection processing unit 65 excludes wall candidates that are located at a distance of at least a first distance threshold in the lateral direction from the first wall candidate, which is designated as the starting point of the wall candidates, in both the left and right directions as viewed from the stereo camera 31. In the illustrated example, the intermittent wall detection processing unit 65 excludes wall candidate DL4, which is located at a distance of at least a first distance threshold in the lateral direction from the first wall candidate DL1, which is designated as the starting point of the intermittent wall, from among the wall candidates DL1 to DL4 created to the left of the vehicle 1. Furthermore, the intermittent wall detection processing unit 65 selects the second wall candidate DL2, which is located less than a first distance threshold in the lateral direction and less than a second distance threshold in the depth direction from the first wall candidate DL1, from among the remaining wall candidates DL2 to DL3, as a target for integration.

[0078] If a second wall candidate DL2 to be integrated exists, the intermittent wall detection processing unit 65 then uses the second wall candidate DL2 as a reference to exclude wall candidates located at a distance of a first distance threshold or more in the lateral direction from the second wall candidate DL2, and searches for wall candidates located at a distance less than the first distance threshold in the lateral direction and less than the second distance threshold in the depth direction from the second wall candidate DL2. The intermittent wall detection processing unit 65 repeats the above search until there are no more wall candidates to be integrated. Similarly, for wall candidates DR1 to DR3 obtained to the right of the vehicle 1, the intermittent wall detection processing unit 65 excludes wall candidate DR3 located at a distance of a first distance threshold or more in the lateral direction from the first wall candidate DR1, and integrates the second wall candidate DR2 that satisfies predetermined conditions with the first wall candidate DR1.

[0079] The lateral or depth distance from the first wall candidate to the other wall candidates may be the distance between the position of the far end of the first wall candidate in the depth direction and the position of the near end of the other wall candidate in the depth direction. However, the intermittent wall detection processing unit 65 may determine the lateral and depth distances based on other positions. Furthermore, the first and second distance thresholds for determining the wall candidates to be merged may be set to any appropriate value.

[0080] Next, the discontinuous wall detection processing unit 65 excludes any wall candidates whose angles differ by more than a predetermined standard from the multiple wall candidates targeted for integration, treating them as noise (step S27). By excluding wall candidates whose angles differ by more than a predetermined standard as noise, the discontinuous wall detection processing unit 65 excludes data that cannot be considered as a series of discontinuous side walls.

[0081] Figure 15 is an explanatory diagram illustrating the process of excluding wall candidates with different angles as noise. The intermittent wall detection processing unit 65 excludes wall candidates with different angles as noise from a plurality of wall candidates targeted for integration in both the left and right directions of the vehicle 1. In the illustrated example, the intermittent wall detection processing unit 65 excludes wall candidate DL5, which is bent outward (further to the left) by, for example, 45 degrees or more relative to the shape of the first wall candidate DL1, which is designated as the starting point of the wall candidates, from among the wall candidates DL1, DL3, and DL5 targeted for integration in the left direction of the vehicle 1, as noise. Also in the illustrated example, the intermittent wall detection processing unit 65 excludes wall candidate DR4, which is bent by, for example, 45 degrees or more relative to the shape of the first wall candidate DR1, which is designated as the starting point of the wall candidates, from among the wall candidates DR1, DR2, and DR4 targeted for integration in the right direction of the vehicle 1, as noise. Note that the angle standard is not limited to 45 degrees and may be set to any appropriate value.

[0082] Next, the intermittent wall detection processing unit 65 generates interpolated wall data by interpolating between adjacent wall candidates from the remaining wall candidates after noise has been removed (step S29). The intermittent wall detection processing unit 65 integrates this interpolated wall data with the multiple wall candidates targeted for integration, making the intermittent walls available for use in driver assistance processing as information about a single continuous side wall.

[0083] Figure 16 is an explanatory diagram illustrating the process of generating interpolated wall data. In the illustrated example, the intermittent wall detection processing unit 65 inserts interpolated wall data DI between multiple wall candidates DL1 and DL3 that are to be integrated in the left direction of the vehicle 1. For example, the intermittent wall detection processing unit 65 calculates the slope of the interpolated wall data DI from the position of the far end of the first wall candidate DL1 in the depth direction and the position of the near end of the second wall candidate DL3 in the depth direction, and inserts the interpolated wall data DI. Similarly, the intermittent wall detection processing unit 65 inserts interpolated wall data DI between multiple wall candidates DR1 and DR2 that are to be integrated in the right direction of the vehicle 1. For example, the interpolated wall data DI may be data of an interpolated point group set at a first distance range Lz in the depth direction on a straight line connecting wall candidates DL1 and DL3.

[0084] Next, the intermittent wall detection processing unit 65 generates detection information for intermittent walls in the current calculation cycle (step S31).

[0085] Figure 17 is an explanatory diagram showing the process for generating intermittent wall detection information. The intermittent wall detection processing unit 65 generates intermittent wall detection information from detection points located near the integrated wall candidate and interpolated wall data (hereinafter also referred to as "integrated wall candidate") in order to improve detection performance. First, the intermittent wall detection processing unit 65 excludes detection points other than those located within a predetermined range (inner effective range) Qi on the side closer to the vehicle 1 than the integrated wall candidate DL, and within an outer effective range Qo on the side further from the vehicle 1 across the integrated wall candidate DL. The inner effective range Qi and the outer effective range Qo are each set as ranges of a predetermined distance laterally from the integrated wall candidate DL. The predetermined distances that determine the inner effective range Qi and the outer effective range Qo may be the same or different.

[0086] Next, the intermittent wall detection processing unit 65 sets reference points for each third distance range in the depth direction on the integrated wall candidate DL, according to the processor's resolution. In the illustrated example, the intermittent wall detection processing unit 65 sets reference points with a third distance range of Lz / 4 (2,048 mm), which is one-quarter of the segment's depth length Lz (8,192 mm). Next, the intermittent wall detection processing unit 65 extracts the closest detection point from each reference point as a representative point, associates it with confidence information for the section containing the detection point, and records it as detection information. At this time, if the reference point is a reference point set on the interpolated wall data, there are no detection points within the third distance range in which the reference point is set. For this reason, the intermittent wall detection processing unit 65 sets the reference point as a representative point, sets the confidence level of the representative point to zero, and records it as detection information. Therefore, the detection information for intermittent walls in this calculation cycle is recorded as detection points and interpolation points located near the integrated wall candidate for each third distance range in the depth direction, and confidence information for each detection point and interpolation point.

[0087] Next, the intermittent wall detection processing unit 65 performs interpolation processing using the past intermittent wall information updated in step S13 (step S33). In order to further improve the detection performance, the intermittent wall detection processing unit 65 interpolates the current output information with the past intermittent wall information that has already been recorded.

[0088] Figure 18 shows a flowchart of the interpolation process using past intermittent wall information. First, the intermittent wall detection processing unit 65 updates the position of the representative point of the past intermittent wall information based on the amount of movement of the vehicle 1 between the previous calculation cycle and the current calculation cycle (step S61). The past intermittent wall information is the intermittent wall information updated in step S11, and includes intermittent wall information calculated in the most recent predetermined number of calculation cycles. The amount of movement of the vehicle 1 between the previous calculation cycle and the current calculation cycle can be calculated, for example, based on the vehicle speed and angular velocity information of the vehicle 1. The intermittent wall detection processing unit 65 may calculate the amount of movement of the vehicle 1 by acquiring the vehicle speed and angular velocity information, or it may acquire the information on the amount of movement of the vehicle 1 from the control device 40.

[0089] Figure 19 is an explanatory diagram showing the process of updating the position of representative points of past intermittent wall information based on the amount of movement of vehicle 1. Assume that in the previous calculation cycle t-1, representative point cloud D of intermittent wall information was detected from vehicle 1 located at the first position P(t-1) in the xz two-dimensional plane. If vehicle 1 moved from the first position P(t-1) to the second position P(t) between the previous calculation cycle t-1 and the current calculation cycle t, the intermittent wall detection processing unit 65 transforms the position of the representative point cloud D of intermittent wall information detected in the previous calculation cycle t-1 to a position on the xz two-dimensional plane with reference to vehicle 1 located at the second position P(t) in the current calculation cycle t, and updates the position of the representative point D' of the detected point cloud of intermittent wall information.

[0090] The intermittent wall detection processing unit 65 records information about the z-coordinate position in the xz two-dimensional coordinate system for the current calculation period t, information about the elapsed frame (how many calculation periods ago it is), and information about the x-position difference determination count, which will be described later, associated with the representative point D' of the detected point group of the updated intermittent wall information.

[0091] Next, the intermittent wall detection processing unit 65 updates the determination information used to determine whether or not to interpolate past intermittent wall information (step S63).

[0092] Figure 20 is an explanatory diagram illustrating the process of updating the information for determination. In the example shown in Figure 20, there are four representative points of past intermittent wall information at equal intervals (Lz / 16) within a third distance range (Lz / 4) determined by the processor's resolution. However, the spacing in the depth direction of the positions of the representative points of past intermittent wall information may change depending on the speed of the vehicle 1. The intermittent wall detection processing unit 65 calculates the average value of the x-coordinates of the representative points of past intermittent wall information for each third distance range (Lz / 4). The intermittent wall detection processing unit 65 calculates and records the average value of the x-coordinates of the representative points of past intermittent wall information for each third distance range in which the representative point of the current detection information exists.

[0093] Next, the intermittent wall detection processing unit 65 compares the position of the representative point of the intermittent wall detection information detected in the current calculation cycle with the position of the representative point of past intermittent wall information (step S65). In this embodiment, the intermittent wall detection processing unit 65 compares the x-coordinate of the representative point of the detection information generated in the current calculation cycle with the average value of the x-coordinates of the representative points D' of past intermittent wall information updated in step S61 that are located within the third distance range.

[0094] Figure 21 is an explanatory diagram illustrating the process of comparing representative points of intermittent wall detection information calculated in the current calculation cycle with representative points of past intermittent wall information. The intermittent wall detection processing unit 65 compares the average value of the x-coordinates of past representative points with the x-coordinate position of the current representative point for each third distance range. The intermittent wall detection processing unit 65 counts the number of times the difference in the x-coordinate position in each third distance range exceeds a predetermined threshold. The predetermined threshold may be set to any appropriate value considering the allowable range of error in the detection result. A higher counter value indicates a lower degree of agreement between the intermittent wall detection information detected in the current calculation cycle and past intermittent wall information, and thus lower reliability of the intermittent wall detection information calculated in the current calculation cycle.

[0095] Next, the intermittent wall detection processing unit 65 determines whether or not to use the detection information for the intermittent wall based on the comparison result of step S65 (step S67). For example, the intermittent wall detection processing unit 65 determines to use the detection information for the intermittent wall if the ratio of the counter value to the total number of representative points included in the detection information is less than 50%. The determination criterion based on the comparison result of step S65 may be set to any appropriate criterion.

[0096] If the intermittent wall detection processing unit 65 does not determine that it will use the current intermittent wall detection information (S67 / No), it deletes the intermittent wall detection information data calculated in the current calculation cycle. In this case, the intermittent wall detection processing unit 65 may replace past intermittent wall information with the current intermittent wall information, and may proceed with processing assuming that there is no output of intermittent wall information in the current calculation cycle.

[0097] On the other hand, if the intermittent wall detection processing unit 65 determines that it will use the current intermittent wall detection information (S67 / Yes), it determines whether or not to interpolate using past intermittent wall information (step S69). For example, the intermittent wall detection processing unit 65 determines whether or not there is past intermittent wall information that can be interpolated outside the search range of the current xz two-dimensional plane, and whether or not there is past intermittent wall information that can be interpolated within the search range of the current xz two-dimensional plane.

[0098] Figure 22 is an explanatory diagram illustrating the general process for determining whether or not there is past intermittent wall information that can be interpolated outside the search range of the current xz two-dimensional plane. First, the intermittent wall detection processing unit 65 uses the representative point furthest in the depth direction from the representative points of the current detection information as a base point, and determines whether or not there are past representative points within the interpolation target range based on that base point. If there are past representative points within the interpolation target range, the intermittent wall detection processing unit 65 uses that representative point as the interpolation target and sets it as a new base point, and then determines whether or not there are past representative points within the interpolation target range again. The intermittent wall detection processing unit 65 repeats this determination towards the back in the depth direction until there are no more past representative points.

[0099] Furthermore, the intermittent wall detection processing unit 65 uses the representative point closest to the viewer in the depth direction among the representative points of the current detection information as a base point, and determines whether or not past representative points exist within the interpolation target range based on that base point. If past representative points exist within the interpolation target range, the intermittent wall detection processing unit 65 uses that representative point as the interpolation target and sets it as a new base point, and then determines whether or not past representative points exist within the interpolation target range again. The intermittent wall detection processing unit 65 repeats this determination toward the viewer in the depth direction until there are no more past representative points.

[0100] The horizontal range of the interpolation target area may be set to any appropriate value, taking into account the error in the displacement of the intermittent walls. The depth range of the interpolation target area may be set to any appropriate value, depending on the assumed vehicle speed, for example, by half of the first distance range Lz of the segment.

[0101] Figure 23 is an explanatory diagram illustrating the schematic of the process for determining whether or not there are representative points of past intermittent wall information that can be interpolated within the search range of the current xz two-dimensional plane. The intermittent wall detection processing unit 65 determines whether or not there are past representative points within the depth range where the representative point of the intermittent wall detection information detected this time is detected. If there are past representative points within the depth range where the representative point of the detected intermittent wall detection information is detected, the intermittent wall detection processing unit 65 will target those representative points for interpolation. The intermittent wall detection processing unit 65 will not target past representative points that are not within the depth range where the representative point of the detected intermittent wall detection information is detected for interpolation.

[0102] If there are no past representative points to be interpolated, the discontinuous wall detection processing unit 65 does not decide to interpolate using past discontinuous wall information (S69 / No), and uses the detection information generated in step S31 as the discontinuous wall information for the current calculation cycle. On the other hand, if there are past representative points to be interpolated, the discontinuous wall detection processing unit 65 decides to interpolate using past discontinuous wall information (S49 / Yes), and reflects the past discontinuous wall information in the detection information generated in step S31 to make it the discontinuous wall information for the current calculation cycle (step S71).

[0103] Figures 24 and 25 are explanatory diagrams illustrating the general process of reflecting past intermittent wall information into the current intermittent wall detection information. As shown in Figure 24, the intermittent wall detection processing unit 65 excludes representative points from the past that are to be interpolated, specifically those that overlap with the representative points of the current detection information on the xz two-dimensional plane. Then, as shown in Figure 25, the intermittent wall detection processing unit 65 interpolates the remaining past representative points to the representative points of the current detection information, and uses this information along with the approximation line to form the intermittent wall information for the current calculation period. The intermittent wall information includes confidence level information for each representative point. Note that the method for calculating the approximation line is not particularly limited.

[0104] Returning to Figure 3, in step S31 or step S33, after the intermittent wall detection processing unit 65 generates intermittent wall information for the current calculation cycle, the side wall detection processing unit 67 identifies a group of detection points estimated to be side walls from a plurality of detection points projected onto the xz two-dimensional plane and generates side wall information (step S35). Since integrally formed side walls have continuity, the side wall detection processing unit 67 identifies a group of detection points estimated to be side walls if, for example, the group of detection points is plotted linearly, or if the group of detection points is plotted in a regular curve, such as having the same radius of curvature. The side wall detection processing unit 67 may determine that a three-dimensional object identified by the three-dimensional object detection processing unit 63 has a relative speed equal to the speed of the vehicle 1 and has a length of a predetermined amount or more in the depth direction (z direction) as seen from the stereo camera 31 is a side wall, and identify the group of detection points corresponding to the position of the three-dimensional object as the group of detection points estimated to be side walls. Note that the method for identifying the group of detection points estimated to be side walls is not limited to the above example.

[0105] Furthermore, the side wall detection processing unit 67 calculates an approximate line of the detected point group identified as representing a side wall. The method for calculating the approximate line of the detected point group identified as representing a side wall is not particularly limited. In addition, the side wall detection processing unit 67 uses the calculated approximate line as a wall candidate and calculates the reliability of the wall candidate for the side wall. For example, similar to the reliability of the wall candidate for the intermittent wall information described above, the side wall detection processing unit 67 extracts the closest detected point from a reference point set for each third distance range in the depth direction according to the processor's resolution, and calculates the reliability of the said representative point. The side wall detection processing unit 67 records the calculated wall candidate information and the reliability information of each representative point in association with the information of the detected point group identified as representing a side wall, as side wall information. Note that the method for calculating the reliability of the wall candidate for the side wall is not particularly limited as long as it is calculated on the same scale as the reliability of the intermittent wall candidate.

[0106] Next, the output information generation processing unit 69 selects from the intermittent wall information and side wall information to be used to generate output information to be output to the control device 40 (step S36). For example, if the approximate lines of the intermittent wall information and the approximate lines of the side wall information do not exist within the same distance range for each distance range in the depth direction as viewed from the stereo camera 31, the output information generation processing unit 69 selects each of the existing intermittent wall information or side wall information as information to be used to generate output information. On the other hand, if the approximate lines of the intermittent wall information and the approximate lines of the side wall information exist within the same distance range, the output information generation processing unit 69 selects the approximate line with the larger number of representative points with high reliability among the intermittent wall information and the approximate lines of the side wall information as information to be used to generate output information. If the approximate lines of the intermittent wall information and the approximate lines of the side wall information exist within the same distance range, the output information generation processing unit 69 may extract representative points with high reliability from the intermittent wall information and the side wall information respectively, and merge the extracted representative points to use as information to generate output information.

[0107] Next, the output information generation processing unit 69 generates output information based on the selected intermittent wall information and side wall information, and transmits the output information to the control device 40 (step S37). In other words, the output information generation processing unit 69 transmits not only side walls with a predetermined length or longer, but also intermittently existing walls and short walls as output information for side walls to the control device 40. The output information generation processing unit 69 may also use the information of the selected approximation line as output information, or it may use the information of the positions of the selected approximation line and the representative points used to calculate the approximation line as output information. Furthermore, the environment recognition device 50 may transmit not only the output information for side walls, but also the information of three-dimensional objects detected by the three-dimensional object detection processing unit 63 to the control device 40. In addition, the output information generation processing unit 69 may determine whether it is appropriate to execute driving support processing based on the output information, based on the confidence level used to generate the output information, and transmit the determination result to the control device 40. The control device 40, having received the output information, executes driving support processing using the detected output information for side walls and intermittent walls and the information for three-dimensional objects.

[0108] Next, the processing unit 51 determines whether or not the vehicle 1's system is stopped (step S39). If the system is not stopped (S39 / No), the process returns to step S13, and the processing of each of the steps described above is repeated as the next calculation cycle. On the other hand, if the system is stopped (S39 / Yes), the processing unit 51 terminates the process of detecting the side wall.

[0109] As described above, the environmental recognition device 50 according to this embodiment includes at least a three-dimensional object detection processing unit 63 that detects three-dimensional objects based on images captured by the stereo camera 31, and a discontinuous wall detection processing unit 65 that detects discontinuous walls based on the captured images. The intermittent wall detection processing unit 65 has the following configuration: it determines a plurality of detection points by projecting unit blocks consisting of a predetermined number of pixels, which are processing units in the captured image, that are within a predetermined height range from the road surface onto an xz two-dimensional plane consisting of the depth direction and the lateral direction as viewed from the stereo camera 31; it groups together detection points that are within a predetermined distance and determines a quadratic approximation model of the grouped detection point group; it divides the xz two-dimensional plane into first distance ranges in the depth direction, and divides the lateral range in each first distance range so as to include the quadratic approximation model, and determines wall candidate lines for each section, and determines wall candidate information including the detection points and wall candidate lines included in the section; it calculates the reliability of the wall candidate information for each section based on the detection points and wall candidate lines included in the section and the information of three-dimensional objects detected by the three-dimensional object detection processing unit 63; and it generates intermittent wall information based on the wall candidate information and the reliability information.

[0110] Therefore, the environmental recognition device can detect even short or intermittently occurring walls as intermittent wall information, and can output this intermittent wall information, along with its reliability, to the control device that performs driving assistance processing. As a result, even when driving in a residential area, for example, lane keeping control and other controls that prevent the vehicle from deviating from the road can be executed without any sense of incongruity.

[0111] Furthermore, in the environmental recognition device 50 according to this embodiment, the intermittent wall detection processing unit 65 calculates a confidence score for each section based on the number of detection points included in the section and the lateral distance between the projection line of the three-dimensional object, which is obtained by projecting the outline of the three-dimensional object detected by the three-dimensional object detection processing unit 63 onto the xz two-dimensional plane, and the candidate wall line. This makes it possible to calculate a highly accurate confidence score based on the degree of agreement between the positions of the detection point group estimated to be intermittent walls and the three-dimensional object information.

[0112] Furthermore, in the environmental recognition device 50 according to this embodiment, the intermittent wall detection processing unit 65, when multiple candidate wall lines in the above section are continuous, treats the continuous candidate wall line as a single candidate wall. If there are multiple candidate wall walls, it excludes candidate wall lines located at a distance of a first distance threshold or more in the lateral direction from the first candidate wall line located closest to the stereo camera 31, and searches for a second candidate wall line located less than the first distance threshold in the lateral direction and less than the second distance threshold in the depth direction from the first candidate wall line, and generates interpolated wall data connecting the first candidate wall line and the second candidate wall line. As a result, intermittent walls can be treated as continuous walls of a predetermined length, and output information can be generated that enables stable execution of control that suppresses vehicle deviation from the road, such as lane keeping control.

[0113] Furthermore, in the environmental recognition device 50 according to this embodiment, if there are multiple wall candidates, the intermittent wall detection processing unit 65 further excludes wall candidates whose angles differ by more than a predetermined angle threshold as noise, and searches for a second wall candidate. This reduces the risk of recognizing an object that does not constitute an intermittent wall as part of an intermittent wall.

[0114] Furthermore, in the environmental recognition device 50 according to this embodiment, the intermittent wall detection processing unit 65 sets reference points for each third distance range in the depth direction on the integrated first wall candidate, second wall candidate, and interpolated wall data, extracts the detection point closest to each reference point as a representative point for each third distance range in the depth direction, and sets the reliability of the section containing the representative point as the reliability of the representative point. This makes it possible to generate output information using detection points with high reliability, thereby improving the reliability of the intermittent wall detection results.

[0115] Furthermore, in the environmental recognition device 50 according to this embodiment, if the reference point is a reference point set on the interpolated wall data and no detection point exists within the third distance range, the intermittent wall detection processing unit 65 sets the reference point as a representative point and sets the confidence level of the representative point to zero. This lowers the priority of the interpolated representative point and reduces the possibility that a representative point with low confidence will be used as output information in priority over other representative points with high confidence.

[0116] Furthermore, in the environmental recognition device 50 according to this embodiment, the intermittent wall detection processing unit 65 divides the xz two-dimensional plane into first distance ranges in the depth direction and into second distance ranges in the lateral direction, extracts effective segments containing at least one detection point from these unit segments, groups the detection points included in multiple consecutive effective segments, and obtains a quadratic approximation model of the grouped detection point group. This makes it possible to accurately estimate the position of one of the intermittently existing walls.

[0117] Furthermore, in the environmental recognition device 50 according to this embodiment, the intermittent wall detection processing unit 65 selects a predetermined number of detection points from the grouped detection point cloud according to the number of detection points included in the detection point cloud, and generates an evaluation model of a quadratic approximation curve using the least squares method multiple times. Detection points that do not fit any of the generated evaluation models are excluded as outliers to obtain a quadratic approximation model. This reduces the variability of the calculation results and improves the accuracy of estimating the position of one of the intermittently existing walls.

[0118] While preferred embodiments of the present disclosure have been described in detail above with reference to the attached drawings, the present disclosure is not limited to such examples. It is clear to any person with ordinary skill in the art to which the present disclosure pertains that various modifications or alterations may be conceived within the scope of the technical idea set forth in the claims, and these will naturally also be understood to fall within the technical scope of the present disclosure.

[0119] Furthermore, the technology disclosed herein can also be realized as a vehicle equipped with the environmental recognition device described in the embodiments above, an environmental recognition processing method executed by the environmental recognition device, a computer program that causes a computer to function as the environmental recognition device described above, and a non-temporary tangible recording medium on which the computer program is recorded.

[0120] 1: Vehicle 31, 31FL, 31FR: Stereo camera 40: Control device 50: Environmental recognition device 51: Processing unit 53: Memory unit 61: Position derivation processing unit 63: Three-dimensional object detection processing unit 65: Intermittent wall detection processing unit 67: Side wall detection processing unit 69: Output information generation processing unit D: Intermittent wall information

Claims

1. The system comprises: a three-dimensional object detection processing unit that detects three-dimensional objects based on images captured by a stereo camera; and a discontinuous wall detection processing unit that detects discontinuous walls based on the captured images, wherein the discontinuous wall detection processing unit obtains a plurality of detection points by projecting unit blocks consisting of a predetermined number of pixels that serve as processing units in the captured images, where the height from the road surface is within a predetermined range, onto a two-dimensional plane consisting of the depth direction and the lateral direction as viewed from the stereo camera; groups the detection points that are within a predetermined distance from each other and obtains a quadratic approximation model of the grouped detection point group; divides the two-dimensional plane into first distance ranges in the depth direction, and divides the lateral range so that each of the first distance ranges includes the quadratic approximation model, and obtains wall candidate lines for each section, and obtains wall candidate information including the detection points and wall candidate lines included in the section; and calculates the reliability of the wall candidate information for each section based on the detection points and wall candidate lines included in the section and the information of the three-dimensional object detected by the three-dimensional object detection processing unit. An environmental recognition device that generates intermittent wall information based on the aforementioned wall candidate information and the aforementioned reliability information.

2. The environmental recognition device according to claim 1, wherein the intermittent wall detection processing unit calculates the reliability for each section based on the number of detection points included in the section and the lateral distance between the projection line of the three-dimensional object, obtained by projecting the outline of the three-dimensional object detected by the three-dimensional object detection processing unit onto the two-dimensional plane, and the candidate wall line.

3. The intermittent wall detection processing unit, when the wall candidate lines of a plurality of sections are continuous, treats the continuous wall candidate lines as a single wall candidate; when there are a plurality of wall candidates, excludes wall candidates located at a distance of a first distance threshold or more in the lateral direction from the first wall candidate located at the closest position from the stereo camera; searches for a second wall candidate located less than the first distance threshold in the lateral direction and less than the second distance threshold in the depth direction from the first wall candidate; and generates interpolated wall data connecting the first wall candidate and the second wall candidate, as described in claim 1.

4. The environmental recognition device according to claim 3, wherein, if there are multiple wall candidates, the intermittent wall detection processing unit further excludes wall candidates whose angles differ by more than a predetermined angle threshold as noise, and searches for the second wall candidate.

5. The intermittent wall detection processing unit sets reference points for each third distance range in the depth direction on the integrated first wall candidate, second wall candidate, and interpolated wall data, extracts the detection point closest to each of the reference points as a representative point for each third distance range in the depth direction, and takes the reliability of the section containing the representative point as the reliability of the representative point, as described in claim 3 or 4.

6. The intermittent wall detection processing unit, if the reference point is a reference point set on the interpolated wall data and the detection point does not exist within the third distance range, sets the reference point as the representative point and sets the confidence level of the representative point to zero, as described in claim 5.

7. The intermittent wall detection processing unit divides the two-dimensional plane into units according to the first distance range in the depth direction and the units according to the second distance range in the lateral direction, extracts an effective segment containing at least one of the detection points, groups together the detection points included in a plurality of consecutive effective segments, and obtains the quadratic approximation model of the grouped detection point group, according to claim 1.

8. The intermittent wall detection processing unit repeatedly selects a predetermined number of detection points from the grouped detection point group according to the number of detection points included in the detection point group and generates an evaluation model of a quadratic approximation curve by least squares method, and removes detection points that do not fit any of the generated evaluation models as outliers to obtain the quadratic approximation model, as described in claim 7.

9. One or more processors are made to perform the following: to detect a three-dimensional object based on an image captured by a stereo camera; to detect intermittent walls based on the image captured; to obtain a plurality of detection points by projecting unit blocks consisting of a predetermined number of pixels, which are processing units in the image captured, that are within a predetermined height range from the road surface, onto a two-dimensional plane consisting of the depth direction and the lateral direction as viewed from the stereo camera; to group the detection points that are within a predetermined distance from each other and obtain a quadratic approximation model of the grouped detection point group; to divide the two-dimensional plane into first distance ranges in the depth direction, and for each section obtained by dividing the lateral range including the quadratic approximation model in each of the first distance ranges, a candidate wall line is generated for each section, and candidate wall information including the detection points and candidate wall lines included in the section is obtained; to calculate the reliability of the candidate wall information for each section based on the detection points and candidate wall lines included in the section and the information of the detected three-dimensional object; and to generate output information based on the candidate wall information and the reliability information. A non-temporary, tangible recording medium that contains a program to execute something.

Images