Driving assistance device

Abstract

An image feature amount extracted from an image obtained by imaging the surroundings with a camera during traveling is acquired as a current image feature amount, a detection result of a sensor 38 for identifying the current position of a vehicle 2 is acquired as sensor information, and the current position of the vehicle 2 is estimated by using map information 57, the sensor information, and the current image feature amount acquired during traveling in an area. After estimating the current position of the vehicle 2, a more detailed current position of the vehicle is identified by matching a point cloud of feature points stored in the map information 57 within a range corresponding to accuracy of estimation of the current position of the vehicle 2 from the estimated current position of the vehicle 2 with feature points extracted from the image obtained by imaging the surroundings with the camera during traveling in the area.

Description

Driving assistance device

[0001] The present invention relates to a driving assistance device that performs driving assistance for vehicles.

[0002] Conventionally, driving assistance devices that provide various types of driving assistance for vehicles, such as assisting driving operations in various scenes, performing part or all of the driving operations on behalf of the driver, and providing information for driving operations, have been known. For example, in parking assistance, which is one type of the above-mentioned driving assistance, it is known to calculate a travel trajectory for parking and perform guidance and vehicle control so that parking is performed in accordance with the calculated travel trajectory.

[0003] For example, Japanese Patent Laid-Open No. 2021-124898 discloses a technology that stores a travel route when a user manually drives a vehicle to a parking space by executing a teaching travel mode, and then executes an automatic travel mode when parking the vehicle in the same parking space, thereby causing the vehicle to automatically travel along the stored travel route to the parking space.

[0004] Japanese Patent Laid-Open No. 2021-124898 (paragraphs 0029, 0038, 0061)

[0005] Here, in order to accurately move the vehicle along the stored travel route when executing the above automatic travel mode, it is important to accurately identify the current position of the vehicle. In the above Patent Document 1, SLAM (Simultaneous Localization and Mapping) is used as a means for identifying the current position of the vehicle. Specifically, feature points are stored on a map, and the detailed current position of the vehicle is identified by matching feature points detected from images captured by an on-board camera with the feature points stored on the map.

[0006] However, for example, immediately after the start of the automatic travel mode, or when the current position of the vehicle is lost for some reason during automatic travel, it is difficult to predict where on the map the current position of the vehicle is located. Therefore, it is necessary to perform matching with a huge number of feature points stored on the map, and there have been cases where the current position of the vehicle cannot be identified, or even if it can be identified, it takes time to complete the identification.

[0007] Therefore, Patent Document 1 discloses a method for narrowing down the feature points to be matched based on the current position of the vehicle estimated by the dead reckoning unit (means for detecting the vehicle's position using on-board sensors). However, the accuracy of estimating the vehicle's current position varies greatly depending on the method. If the range for narrowing the range is too narrow relative to the estimation accuracy, it may become impossible to determine the vehicle's current position. Conversely, if the range is too wide, it may also become impossible to determine the vehicle's current position, or even if it can be determined, it may take a long time to do so.

[0008] The present invention was made to solve the aforementioned problems of the conventional invention, and aims to provide a driving assistance device that enables accurate and rapid identification of the vehicle's position by pre-generating map information that stores image features extracted from images captured at multiple different imaging positions, estimating the vehicle's position in real time using image features extracted from captured images, sensor information, and the map information, and narrowing down the feature points to be matched according to the accuracy of the estimation.

[0009] To achieve the above objective, the driving assistance device according to the present invention extracts image features from images of the surrounding area captured at multiple different imaging positions by an imaging device installed in the vehicle, generates map information by associating the image features extracted for each image with the imaging position from which the source image was captured, and when the vehicle is traveling in the area where the map information is generated, it acquires the image features extracted from images of the surrounding area captured by the imaging device during the journey as current image features, and acquires the detection results of a sensor for identifying the vehicle's current position as sensor information, estimates the vehicle's current position using the current image features acquired during the journey in the area, the sensor information and the map information, and after estimating the vehicle's current position, it matches a point cloud of feature points stored in the map information within a range corresponding to the accuracy of the estimation of the vehicle's current position with the feature points extracted from images of the surrounding area captured by the imaging device during the journey in the area to determine the vehicle's current position in more detail. Furthermore, the term "sensor" is not limited to sensors that can directly measure the vehicle's current position, such as GPS, but also includes sensors that can measure the vehicle's current position relative to its past position, such as vehicle speed sensors, steering sensors, and gyro sensors.

[0010] According to the driver assistance device of the present invention having the above configuration, map information is generated in advance by storing image features extracted from images captured at multiple different imaging positions, while the vehicle's position is estimated in real time using image features extracted from the captured images, sensor information, and the map information. This makes it possible to estimate the vehicle's current position to some extent before matching feature points, for example, at the initial timing of determining the vehicle's current position immediately after the start of assistance, or even when the vehicle's current position is lost for some reason. Furthermore, by narrowing down the feature points to be matched according to the accuracy of the estimation, it becomes possible to narrow down the range to an appropriate extent, enabling accurate and rapid identification of the vehicle's position.

[0011] This is a schematic diagram of the vehicle according to this embodiment. This is a block diagram showing the configuration of the driver assistance device according to this embodiment. This is a flowchart of the parking assistance information registration processing program according to this embodiment. This is a diagram showing the display screen shown on the liquid crystal display while driving. This is a diagram showing an example of feature points extracted from captured images. This is a diagram showing an example of generated map information. This is a diagram explaining the image features stored in the map information. This is a flowchart of the parking assistance processing program according to this embodiment. This is a diagram showing an example of the parking execution selection screen displayed on the liquid crystal display. This is a diagram explaining the method for estimating the vehicle's current position using image features.

[0012] Hereinafter, one embodiment of the driver assistance device according to the present invention will be described in detail with reference to the drawings. First, the vehicle 2 equipped with the driver assistance device 1 according to this embodiment will be described below. Figure 1 is a schematic diagram of the vehicle 2 according to this embodiment.

[0013] Here, Vehicle 2 may be, for example, an automobile powered by an internal combustion engine (internal combustion engine vehicle), an automobile powered by an electric motor (electric vehicle, fuel cell vehicle, etc.), or an automobile powered by both (hybrid vehicle). Furthermore, there is no restriction on the type of vehicle; it may be a regular passenger car, a large commercial truck, a bus, construction machinery, etc. Also, although the following explanation will refer to it as a four-wheeled vehicle, it may also be a two-wheeled or three-wheeled vehicle.

[0014] However, Vehicle 2 shall be a vehicle capable of not only manual driving based on the user's driving operations, but also assisted driving through automated driving assistance, in which the vehicle drives automatically without user operation.

[0015] Furthermore, autonomous driving assistance may be performed only under specific circumstances, such as when parking or exiting a parking space, or it may be performed on all road sections, or it may be configured to be performed only while the vehicle is traveling on a specific road section (for example, a highway with a gate (regardless of whether it is manned or unmanned, tolled or free) at the boundary). In the following explanation, the autonomous driving sections in which the vehicle's autonomous driving assistance is performed will include all road sections, including general roads and highways, as well as parking lots, and will only be performed when the user has selected to perform autonomous driving assistance (for example, by turning on the autonomous driving start button) and it has been determined that it is possible to perform driving with autonomous driving assistance. On the other hand, vehicle 2 may be a vehicle that is only capable of driving with autonomous driving assistance. Alternatively, autonomous driving assistance may be performed only when the vehicle is driving to a parking space (i.e., parking assistance).

[0016] In the vehicle control of the automated driving assistance system of this embodiment, for example, the current position of the vehicle, the lane the vehicle is traveling in, and the position of surrounding obstacles are detected in real time, and vehicle control such as steering, drive source, and brakes is automatically performed so that the vehicle travels along the generated driving trajectory at a speed according to the generated speed plan. In particular, when providing parking assistance, the system uses the detection results of sensors and cameras to confirm the parking space to which the vehicle will park and the surrounding conditions, calculates the parking trajectory to the parking space, and automatically performs vehicle control to enter the parking space along the calculated parking trajectory and complete parking. Furthermore, in addition to the above-mentioned normal parking assistance (parking assistance to nearby parking positions), it also supports long-range parking assistance, which targets a predetermined distant parking position, such as a home garage or a monthly contracted parking space in a parking lot. In long-range parking assistance, as described later, the parking trajectory to the parking space is registered in advance, and vehicle control is automatically performed to travel along the registered parking trajectory and complete parking. However, in the above-mentioned parking assistance, only the steering operation may be automated, and the control of the drive source and brakes may be based on manual operation. Alternatively, the system could only provide guidance on parking paths to parking spaces or vehicle operation, leaving the actual parking operation to the user manually.

[0017] As shown in Figure 1, the vehicle 2 includes an operating unit 3 that receives input from the occupant, a liquid crystal display 4 that displays information related to driving assistance to the occupant, a speaker 5 that outputs voice guidance related to driving assistance, a front camera 6, a rear camera 7, and side cameras 8A and 8B for imaging the area around the vehicle, ultrasonic sensors 9A to 9L that detect obstacles around the vehicle, and a driving assistance ECU (Electronic Control Unit) 10 that performs various calculations based on the input information. The driving assistance device 1 includes the above-mentioned driving assistance ECU 10.

[0018] The following describes the various components of the vehicle 2. First, the control unit 3 is located, for example, in front of the steering wheel and includes control buttons that are operated when starting the automatic driving assistance. By operating the control unit 3, the user can switch between manual driving, where the vehicle drives based on the user's driving input, and assisted driving with automatic driving assistance, where the vehicle drives automatically without user input. In this embodiment, the user also uses the control unit 3 to register parking trajectories for long-range parking. The control unit 3 may also have a touch panel located in front of the liquid crystal display 4. It may also have a microphone and a voice recognition device.

[0019] The liquid crystal display 4 is a type of display device installed on the instrument panel of the vehicle 2. For example, it displays map information of the area around the vehicle's current location, and when parking assistance is performed, it displays bird's-eye view and overhead view images of the area around the vehicle, which are generated by processing and combining images captured by the front camera 6, rear camera 7, and side cameras 8A and 8B. In addition, if there are warning objects such as pedestrians around the vehicle 2, a warning image indicating the presence of the warning object may also be displayed at a position between the vehicle 2 and the warning object in the overhead view and bird's-eye view images. The liquid crystal display 4 may also be used for the navigation system.

[0020] Furthermore, speaker 5 is mounted on the instrument panel of vehicle 2 and outputs voice guidance and warning sounds related to driver assistance. Speaker 5 may also be used for the navigation system.

[0021] Furthermore, the front camera 6 is an imaging device that has a camera using a solid-state image sensor such as a CCD, and is installed, for example, above the front bumper of the vehicle 2 or behind the rearview mirror, with the optical axis facing forward in the direction of travel of the vehicle.

[0022] The rear camera 7 is an imaging device that also has a camera using a solid-state image sensor such as a CCD, and is mounted, for example, near the center above the license plate attached to the rear of the vehicle 2, with the optical axis facing the rear of the vehicle.

[0023] Furthermore, the side cameras 8A and 8B are imaging devices that also have cameras using solid-state image sensors such as CCDs, and are mounted, for example, on the left and right side mirrors of the vehicle 2, with the optical axis facing the side of the vehicle.

[0024] The driver assistance ECU 10 then performs image recognition processing on the images captured by the front camera 6, rear camera 7, and side cameras 8A and 8B during the execution of automated driving assistance. This process detects lane markings, parking space markings, and obstacles (other vehicles, pedestrians, bicycles, walls, guardrails, and other structures) around the vehicle, and executes automated driving assistance based on the detection results. It also performs vehicle position determination using Visual SLAM (Simultaneous Localization and Mapping) with the captured images. Furthermore, when registering parking trajectories for long-range parking, as described later, it also extracts image features by feature-generating the images captured by each camera.

[0025] Meanwhile, the ultrasonic sensors 9A to 9L are arranged at predetermined intervals on the front, rear, and sides of the vehicle, respectively. They transmit ultrasonic waves as probe waves around the vehicle 2 and detect objects that reflected the probe waves by receiving reflected waves from objects around the vehicle. Specifically, they are a type of distance measuring sensor capable of detecting the distance (measured distance value) to the object that reflected the probe waves by measuring the time from transmission to reception. Furthermore, the ultrasonic sensors 9A to 9L are configured to generate an output signal (including the distance to the detected object) corresponding to the reception result of the received wave and output it to the control unit. The objects to be detected by the ultrasonic sensors 9A to 9L include, for example, people, bicycles, other vehicles, walls, and other obstacles that the vehicle 2 needs to avoid while driving. In addition, millimeter-wave sensors or laser sensors may be used as distance measuring sensors instead of ultrasonic sensors.

[0026] Furthermore, while the installation position and direction of each ultrasonic sensor 9A to 9L can be set as appropriate, in this embodiment, in order to make the detection range of the target object encompass all directions in front of, behind, and to the left and right of the vehicle's direction of travel, for example, ultrasonic sensors 9A to 9D are installed on the front of the vehicle 2 facing the direction of travel so that the direction of transmission of the probe wave is in front of the vehicle's direction of travel. Ultrasonic sensors 9E and 9F are installed on the left side of the vehicle 2 facing left so that the direction of transmission of the probe wave is to the left of the vehicle's direction of travel. Ultrasonic sensors 9G and 9H are installed on the right side of the vehicle 2 facing right so that the direction of transmission of the probe wave is to the right of the vehicle's direction of travel. Ultrasonic sensors 9I to 9L are installed on the rear of the vehicle 2 facing the opposite direction of travel so that the direction of transmission of the probe wave is to the rear of the vehicle. The height of each ultrasonic sensor 9A to 9L from the ground surface is approximately the same.

[0027] In this embodiment, among the ultrasonic sensors 9A to 9L, the ultrasonic sensors 9A to 9D on the front of the vehicle 2 and the ultrasonic sensors 9I to 9L on the rear of the vehicle 2 are installed in positions where they can receive reflected waves as indirect waves from adjacent sensors. By receiving both direct and indirect waves, it is possible to determine not only the distance to the object but also the specific position of the object (relative position to the vehicle) using triangulation. The ultrasonic sensors 9E to 9H on the sides are installed spaced apart from each other and therefore cannot receive indirect waves. However, as the vehicle moves, it is possible to similarly determine the specific position of the object (relative position to the vehicle) using triangulation with respect to the distance measured at the previous position, the distance measured at the current position, and the distance traveled between them.

[0028] On the other hand, the driver assistance ECU 10 is an electronic control unit that performs various processes related to automated driving assistance. For example, it continuously detects the vehicle's current position, the lane the vehicle is traveling in, and the positions of surrounding obstacles, and controls the vehicle, such as steering, drivetrain, and brakes, to ensure that the vehicle travels along a generated driving trajectory at a speed according to a similarly generated speed plan. In particular, when performing normal parking assistance to park in a nearby parking space, it uses the detection results from the aforementioned front camera 6, rear camera 7, side cameras 8A, 8B, and ultrasonic sensors 9A to 9L to confirm the parking space and its surroundings, calculates a parking trajectory to the parking space, and controls the vehicle to enter the parking space along the calculated parking trajectory and complete the parking. On the other hand, when performing parking assistance for long-range parking to a distant parking space, it acquires a parking trajectory to move to a parking space (including movement on private roads and within the property) that has been registered in advance, such as a home garage or a monthly contracted parking space in a parking lot, and controls the vehicle to enter the parking space along the acquired parking trajectory and complete the parking. The driver assistance ECU 10 is connected to the aforementioned control unit 3, LCD display 4, speaker 5, front camera 6, rear camera 7, side cameras 8A, 8B, and ultrasonic sensors 9A to 9L via an in-vehicle network such as CAN. It is also connected to various sensors mounted on the vehicle 2, such as GPS, vehicle speed sensor, wheel speed sensor, acceleration sensor, gyro sensor, steering sensor, and shift position sensor, as well as in-vehicle devices such as the navigation system. The detailed configuration of the driver assistance ECU 10 will be described later.

[0029] In addition to the components shown in Figure 1, Vehicle 2 also has other basic components as a vehicle; however, only the configuration related to the control of the automated driving assistance system and the control related to said configuration will be explained.

[0030] Next, we will describe in detail the driver assistance ECU 10, which is part of the driver assistance system 1 provided by the vehicle 2 described above. Figure 2 is a block diagram showing the configuration of the driver assistance system 1 according to this embodiment.

[0031] As shown in Figure 2, the driver assistance ECU (Electronic Control Unit) 10 is an electronic control unit that controls the entire driver assistance system 1. It includes a CPU 31 as a calculation device and control device, a RAM 32 which is used as working memory when the CPU 31 performs various calculations and stores driving trajectory data when the driving trajectory is calculated, a ROM 33 which stores control programs as well as the parking assistance information registration processing program (see Figure 3) and parking assistance processing program (see Figure 8) described later, and a flash memory 34 which stores programs read from the ROM 33. The driver assistance ECU 10 executes various functions as a processing algorithm. For example, the system may have functions such as: extracting image features from images taken at multiple different imaging positions by a camera on the vehicle, generating map information by associating the extracted image features with the imaging position from which the original image was taken; acquiring image features extracted from images taken by the camera while the vehicle is driving as current image features, and acquiring the detection results of sensors to determine the vehicle's current position as sensor information; estimating the vehicle's current position using the current image features, sensor information, and map information acquired while driving in the area; and, after estimating the vehicle's current position, matching a point cloud of feature points stored in the map information within a range corresponding to the accuracy of the vehicle's current position estimation with feature points extracted from images taken by the imaging device while driving in the area to determine the vehicle's current position in more detail.

[0032] Furthermore, the driver assistance ECU 10 is connected to various sensors 38 for detecting the vehicle's current position and behavior, such as GPS, vehicle speed sensor, wheel speed sensor, acceleration sensor, gyro sensor, steering sensor, and shift position sensor, as well as to various drive units 39 of the vehicle, such as steering, brakes, accelerator, and transmission. Based on the detection results of these sensors 38, the ECU detects the vehicle's current position and behavior and controls each drive unit 39 to provide automatic driving assistance for the vehicle 2. In particular, in this embodiment, the detailed current position of the vehicle can be determined by Visual SLAM, and the detection results of the sensors 38 are used to narrow down the feature points to be matched.

[0033] Furthermore, ROM33 includes vehicle information DB35. Vehicle information DB35 stores various information about vehicle 2. For example, it stores the installation positions (height from the ground, left-right position) and detection axes (optical axis for cameras) of cameras and ultrasonic sensors 9A to 9L installed on vehicle 2, as well as the overall length, vehicle width, wheelbase, and minimum turning radius. This information is entered in advance by the occupants or personnel from the vehicle manufacturer.

[0034] On the other hand, the flash memory 34 includes a generated map information DB 36. The generated map information DB 36 stores the map information (environmental map) generated in Visual SLAM. The map information is generated for areas that the vehicle has traveled in the past, and is a three-dimensional map information in which a cloud of feature points is placed (mapped) in three-dimensional space. The feature points are extracted by performing a feature point extraction process on the images captured by the front camera 6, rear camera 7, and side cameras 8A and 8B while the vehicle is in motion. Specifically, distinctive points in the image (such as corners and areas where brightness changes) become feature points. Further details of the map information will be described later.

[0035] In this embodiment, image features are also stored for the three-dimensional map information described above. The image features are stored in combination with the imaging location, and the image features of the images captured by the vehicle at that imaging location are stored. In particular, image features are stored for multiple different imaging locations on the map, and by comparing them with image features extracted from images captured in real time, as described later, it is possible to estimate the vehicle's current position. Image features are data that shows the characteristics of an image by compressing the entire image, for example, a 4096-dimensional multidimensional vector, that is, data that shows the characteristics of the image by defining 4096 types of features by the length and direction of the vector. An existing machine learning algorithm is used to extract image features from captured images.

[0036] Furthermore, the flash memory 34 also includes a registration information DB 37. The registration information DB 37 is a storage means that stores parking assistance information for long-range parking that has been registered by the user up to the present. The parking assistance information stored in the registration information DB 37 is information that specifies what kind of parking assistance to perform when parking assistance is performed, and in particular includes the parking trajectory (driving trajectory) and parking position when parking. In addition to the parking trajectory and parking position, it may also include the parking direction. The parking assistance information may also be stored as part of the map information stored in the generated map information DB 36. That is, the parking trajectory and parking position may be stored in the map information. For example, when a vehicle approaches a registered parking position, the user is asked whether or not to perform long-range parking assistance, and if the user chooses to perform it, parking assistance is performed that automatically drives the vehicle according to the parking trajectory included in the parking assistance information.

[0037] Furthermore, the parking assistance information stored in the registration information DB37 is registered when the user manually drives to the parking position after performing a predetermined operation using the operation unit 3, as described later. Note that there is an upper limit (for example, 20) on the number of parking assistance information entries that can be registered in the registration information DB37. If the number of entries exceeds this limit, the user can select and overwrite unnecessary parking assistance information. It is also possible to use a removable memory card, for example, as the storage medium for the registration information DB37.

[0038] Next, the parking assistance information registration processing program executed by the driver assistance ECU 10 in the driver assistance device 1 having the above configuration will be explained with reference to Figure 3. Figure 3 is a flowchart of the parking assistance information registration processing program according to this embodiment. Here, the parking assistance information registration processing program is executed after the ACC power (accessory power supply) of the vehicle 2 is turned ON, and is executed when a predetermined operation to start the registration of parking assistance information for long-range parking is received. In addition to registering parking assistance information, it is a program that generates map information in which feature points and image features are stored. The programs shown in the flowcharts in Figures 3 and 8 below are stored in the RAM 32 and ROM 33 of the driver assistance device 1 and are executed by the CPU 31.

[0039] First, in step 1 (hereinafter abbreviated as S), the CPU 31 determines whether or not the operation unit 3 has received a predetermined operation to start registering parking assistance information for long-range parking. For example, as shown in Figure 4, the liquid crystal display 4 displays a map image 51 of the area around the vehicle while the vehicle is in motion (excluding during parking assistance), and a registration start icon 52 is also displayed along with the map image 51. When a user wishes to start registering parking assistance information for long-range parking while the vehicle is in motion, they perform the operation to select the registration start icon 52. In addition, when a user wishes to start registering parking assistance information for long-range parking, for example, when their home or workplace has changed and they want to register a new parking location or parking trajectory as parking assistance information when performing long-range parking assistance. In addition, while registering parking assistance information for long-range parking, the vehicle is basically driven by the user's manual driving.

[0040] Then, if the operation unit 3 determines that it has received a predetermined operation to start registering parking assistance information for long-range parking (S1: YES), the process proceeds to S2. On the other hand, if the operation unit 3 determines that it has not received a predetermined operation to start registering parking assistance information for long-range parking (S1: NO), the parking assistance information registration processing program is terminated.

[0041] The following processes from S2 onward will be repeatedly executed at predetermined time intervals (for example, every 250 ms) until the vehicle completes parking (S9: YES).

[0042] First, in S2, the CPU 31 determines whether the vehicle has moved a predetermined distance X since the processing in S3 and S4. The value of the predetermined distance X can be set as appropriate, but for example, it can be set to a distance between a few centimeters and several tens of centimeters. Setting the predetermined distance X to a small value will increase the accuracy of the vehicle position estimation described later, but setting it too small will increase the data size of the generated map information and increase the processing burden, so it is desirable to set it to an appropriate distance. The distance the vehicle has moved can be measured using the detection result of the vehicle speed sensor installed in the vehicle.

[0043] Then, if it is determined that the vehicle has traveled a predetermined distance X after the previous processing of S3 and S4 (S2: YES), the process proceeds to S3. In contrast, if it is determined that the vehicle has not traveled the full predetermined distance X after the previous processing of S3 and S4 (S2: NO), the process proceeds to S5.

[0044] In S3, the CPU 31 respectively acquires the captured images most recently captured by the front camera 6, the rear camera 7, and the side cameras 8A and 8B. In the present embodiment, the captured images from all four-direction cameras are acquired; however, the configuration may be adapted to acquire captured images from some of the cameras (for example, only the rear camera 7).

[0045] Next, in S4, the CPU 31 extracts image feature quantities from the captured images acquired in said S3. Here, the image feature quantities are data representing the characteristics of an image obtained by compressing the entire image. For example, the image feature quantities are data representing the characteristics of the image that are defined as a 4096-dimensional multidimensional vector, that is, 4096 types of features defined by the magnitude and direction of the vector. For example, the CPU 31 extracts image feature quantities by converting a captured image into feature quantities using an existing algorithm based on machine learning. However, the number of dimensions of the image feature quantities is not necessarily required to be 4096, and the image feature quantities may be extracted with a larger or smaller number of dimensions.

[0046] Then, the processing of said S3 and S4 is executed every time the vehicle travels the predetermined distance X. That is, while registering parking assistance information for long-range parking, every time the vehicle advances the predetermined distance X, an image feature quantity is extracted from the captured image taken at that position.

[0047] Thereafter, in S5, the CPU 31 performs feature point extraction processing on the captured images most recently captured by the front camera 6, the rear camera 7, and the side cameras 8A and 8B. That is, in the present embodiment, feature point extraction processing is repeatedly performed from captured images captured in real time at predetermined time intervals (e.g., every 250 ms). A feature point is a point having characteristics in an image (such as a corner or a location where brightness changes), and as a method for extracting feature points from an image, there are existing algorithms using machine learning such as the SIFT (Scale-Invariant Feature Transform) algorithm, for example. For example, FIG. 5 is a diagram showing an example of feature points 56 extracted from a captured image 55 captured by the rear camera 7. As shown in FIG. 5, for example, the boundary between asphalt and a lane marking where brightness changes greatly, the boundary between a building and the background, the corner of a structure, and the like are extracted as the feature points 56.

[0048] It should be noted that in the present embodiment, feature points are extracted from captured images of all cameras in four directions, but feature points may be extracted from captured images of some cameras (for example, only the rear camera 7).

[0049] Thereafter, in S6, the CPU 31 generates three-dimensional map information in which the point cloud of the feature points 56 extracted in S5 is arranged (mapped) in a three-dimensional space. It should be noted that although the distance from the vehicle is unknown for the feature points 56 extracted from a single image, the distance to the feature points 56 can be calculated from the state of change of the feature points 56 extracted in successive captured images before and after, and the positions of the feature points 56 in the three-dimensional space can be calculated. Since the positions of the feature points 56 are calculated based on the position of the vehicle 2, for example, with the initial position of the vehicle as the origin, feature point matching is performed in S7 and S8 described later to specify the current position and posture of the vehicle, and the generation of map information in S6 is also performed in parallel.

[0050] As a result, as shown in Figure 6, three-dimensional map information 57 is generated with feature points 56 placed (mapped). The feature points 56 placed in the map information 57 indicate, for example, the boundary between asphalt and road markings, the boundary between buildings and the background, the corners of structures, etc., at their respective locations. In other words, the map information 57 does not contain information to directly identify roads or structures, like map information used for route guidance, but it is map information that identifies how roads and structures exist through the placement of feature points 56. The map information 57 is generated targeting the area around the travel trajectory 60 that the vehicle traveled (within the camera's imaging range). The map information 57 shown in Figure 6 is a plan view because it is viewed from vertically above (in the z-axis direction), but in reality it also contains information in the height direction.

[0051] Furthermore, in S7, the CPU 31 matches the feature points 56 extracted from the images captured most recently by the front camera 6, rear camera 7, and side cameras 8A and 8B with the point cloud of feature points 56 placed in the map information 57 generated in 6. Specifically, matching is performed by comparing the feature quantities of the feature points and associating feature points with high similarity. For example, methods such as brute-force matching and cross-checking are used. The point cloud of feature points to be matched is narrowed down based on the current position and orientation of the vehicle that has been identified so far. That is, from the point cloud of feature points stored in the map information, the range of the point cloud of feature points to be matched next can be predicted from the current position and orientation of the vehicle that has been identified so far, so matching is performed on the point cloud of feature points stored in the map information within the predicted range. By narrowing down the range of the point cloud of feature points to be matched in advance, the speed of identifying the current position can be improved and the current position can be identified more accurately and reliably.

[0052] Then, in S8, the CPU 31 identifies the new vehicle's current position and orientation (vehicle orientation) based on the matching results of S8. Based on the identified vehicle's current position and orientation, the map information generation in S6 is further performed.

[0053] Furthermore, the method of extracting feature points 56 from images captured by a camera as described in S5 to S8 above, generating map information 57 by placing the point cloud of the extracted feature points 56, and then using the generated map information 57 and the feature points 56 extracted from the captured images in real time to determine the current position and orientation of the vehicle is called Visual SLAM. However, LiDAR SLAM or Depth SLAM can also be used as methods for determining the position and orientation of a vehicle using feature points.

[0054] Furthermore, the map information generated in S6 also stores the vehicle's travel path 60. Specifically, the line segment connecting the history of the vehicle's current position identified in S8 becomes the vehicle's travel path 60.

[0055] In this embodiment, the image features extracted in S4 are also stored in addition to the map information generated in S6. Specifically, the image features are stored in association with the imaging position where the source image was captured. The imaging position is determined based on the result of identifying the vehicle's current position in S8. In this embodiment, as described above, image features are extracted each time the vehicle moves a distance X (S4), so as shown in Figure 7, the image features are stored in association with imaging positions located at distance X intervals on the travel track 60. In the example shown in Figure 7, image feature A is stored for imaging position A on the travel track 60, image feature B is stored for imaging position B, image feature C is stored for imaging position C, and image feature D is stored for imaging position D.

[0056] Next, in S9, the CPU 31 determines whether or not the vehicle has completed parking. Specifically, it determines that the vehicle has completed parking if the vehicle speed is 0 and the shift position is 'P'.

[0057] If it is determined that the vehicle has been parked (S9: YES), the process proceeds to S10. Conversely, if it is determined that the vehicle has not been parked (S9: NO), the process returns to S2.

[0058] Subsequently, in S10, the CPU 31 acquires the driving path of the vehicle from the time the operation to start registering parking assistance information for long-range parking is initiated until parking is completed, as the parking path. Specifically, the line segment connecting the history of the vehicle's current position identified in S8 is acquired as the parking path. The vehicle's position at the time parking is completed is also acquired as the parking position. The acquired parking path and parking position are then registered in the registration information DB 37 as parking assistance information for long-range parking. Note that the parking assistance information may also be included as part of the map information 57 generated in S6. That is, the parking path and parking position may be stored in the map information 57 generated in S6. Furthermore, the parking direction may also be acquired and registered as parking assistance information.

[0059] Next, the parking assistance processing program executed by the driver assistance ECU 10 in the driver assistance device 1 having the above configuration will be described with reference to Figure 8. Figure 8 is a flowchart of the parking assistance processing program according to this embodiment. Here, the parking assistance processing program is executed after the ACC power (accessory power supply) of the vehicle 2 is turned ON, and is a program that provides parking assistance, especially when the vehicle is parked, as one of the automatic driving assistance functions. In the following description, parking assistance when the vehicle is performing long-range parking will be described.

[0060] Here, long-range parking refers to parking aimed at a predetermined parking space located far away, such as a home garage or a monthly-contracted parking space in a parking lot, and presupposes a relatively long distance to the target parking space. Furthermore, the following explanation assumes that at least one parking assistance program for long-range parking has been registered by the aforementioned parking assistance information registration processing program (Figure 3), and that the user wishes to park the vehicle using one of the registered parking assistance programs. In addition, the vehicle may be driving with autonomous driving assistance or manually before starting parking assistance.

[0061] First, in S11, the CPU 31 acquires the vehicle's current position using various sensors such as GPS and a vehicle speed sensor.

[0062] Next, in S12, the CPU 31 reads the parking assistance information for long-range parking registered in the registration information DB 37 and determines whether or not there is parking assistance information in which a parking position is registered within a predetermined distance (for example, 1 km) from the vehicle's current position. If the user's destination (corresponding to the place where the user wishes to park the vehicle) is known, the determination process in S12 may be performed only on parking assistance information in which a parking position is registered around the destination.

[0063] If it is determined that there is parking assistance information with a registered parking location within a predetermined distance from the vehicle's current location (S12: YES), the process proceeds to S13. Conversely, if it is determined that there is no parking assistance information with a registered parking location within a predetermined distance from the vehicle's current location (S12: NO), the process proceeds to S11.

[0064] In S13, the CPU 31 displays a parking execution selection screen 71 that asks the user whether or not to start long-range parking assistance. Figure 9 shows the parking execution selection screen 71 displayed on the liquid crystal display 4. If other information is displayed on the liquid crystal display 4 when the display conditions for the parking execution selection screen 71 are met, the other information will continue to be displayed while the parking execution selection screen 71 is newly displayed on a portion of the liquid crystal display 4 screen. For example, in the example shown in Figure 9, the display conditions for the parking execution selection screen 71 are met when a map image 51 of the area around the vehicle is displayed on the liquid crystal display 4, and the display of the map image 51 will continue while the parking execution selection screen 71 is displayed.

[0065] As shown in Figure 9, the parking execution selection screen 71 displays a message asking whether or not to start long-range parking assistance, as well as an icon for the user to select whether or not to start long-range parking assistance. At that time, the user may also be guided about the content of the parking assistance to be started (parking trajectory and parking position). The user selects the YES icon if they wish to start long-range parking assistance, and the user selects the NO icon if they do not wish to receive long-range parking assistance.

[0066] However, the system may proceed to S15 when the conditions for determination in S12 are met, and the long-range parking assistance may be automatically started.

[0067] Then, if the user selects to start long-range parking assistance on the parking execution selection screen 71 (S14: YES), the program proceeds to S15. Conversely, if the user does not select to start long-range parking assistance (S14: NO), the parking assistance processing program is terminated.

[0068] From S15 onward, the CPU 31 performs long-range parking assistance according to the parking assistance information that satisfies the determination conditions of S12. Specifically, it reads the corresponding parking assistance information from the registered information DB 37, reads the corresponding map information 57 from the generated map information DB 36, and automatically controls the vehicle to drive along the parking trajectory included in the parking assistance information and complete parking. Since map information 57 has been generated (S6) for the area surrounding the parking trajectory registered by the aforementioned parking assistance information registration processing program (Figure 3), it is assumed that the vehicle will drive in an area where map information 57 has already been generated by the parking assistance information registration processing program (Figure 3).

[0069] The following processes from S15 onward are executed repeatedly at predetermined time intervals (for example, every 250 ms) until the vehicle completes parking (S26: YES).

[0070] First, the CPU 31 determines whether it is the first time to execute the process from S15 onwards after starting the parking assistance for long-range parking by the parking assistance processing program, that is, whether it is immediately after the start of parking assistance.

[0071] Then, if it is determined that the process from S15 onwards will be executed for the first time since the start of long-range parking assistance (S15: YES), the process proceeds to S16. On the other hand, if it is determined that the process from S15 onwards has already been executed at least once since the start of long-range parking assistance (S15: NO), the process proceeds to S31.

[0072] In S16, the CPU 31 acquires real-time images captured by the front camera 6, rear camera 7, and side cameras 8A and 8B as the vehicle 2 moves along the parking track. In this embodiment, images from all four cameras are acquired, but images from only some cameras (for example, only the rear camera 7) may be acquired.

[0073] Next, in S17, the CPU 31 extracts image features from the captured image acquired in S16. The details are the same as in S4, so the explanation will be omitted. The image features extracted in S17 will be referred to as image features from now on.

[0074] Next, in S18, the CPU 31 reads the corresponding map information 57 (the map information generated in S6 when registering the parking track and parking location that are the target of this parking assistance) from the generated map information DB 36, and compares (feature matching) the current image features extracted in S17 with the image features stored in the map information 57 (hereinafter referred to as registered image features). Here, as shown in Figure 7, the map information 57 stores registered image features associated with imaging positions located at distance X intervals on the driving track 60 (corresponding to the parking track on which the vehicle is currently traveling). Therefore, in S18, the current image features extracted in S17 are compared with each of the multiple registered image features stored in the map information (for example, image features A to D in the example shown in Figure 7). In this embodiment, since the image features are represented by 4096-dimensional multidimensional vectors, it is possible to evaluate the similarity between the current image features and the registered image features by comparing the length and direction of each vector between them.

[0075] Subsequently, in S19, the CPU 31 selects a registered image feature that is closest to the current image feature based on the comparison result in S18. That is, from among the multiple registered image feature quantities stored in the map information, it selects the registered image feature whose vector lengths and directions are most similar to the current image feature.

[0076] Furthermore, in S20, the CPU 31 acquires the detection results of the GPS installed in the vehicle 2 as sensor information via CAN, and determines the vehicle's current position from the acquired sensor information. The vehicle 2 is equipped with GPS, vehicle speed sensor, wheel speed sensor, acceleration sensor, gyro sensor, steering sensor, etc., as sensors for determining its current position, but since the GPS can determine the vehicle's position in absolute terms, it is possible to determine the vehicle's current position even immediately after the system starts. However, the accuracy of GPS current position detection is not high, and for example, it can only determine the current position within a range of a few meters.

[0077] Next, in S21, the CPU 31 determines whether the imaging position associated with the registered image feature selected in S19 matches the current position of the vehicle identified in S20 based on sensor information. As described above, the current position of the vehicle identified by GPS is within a range of several meters, so if the imaging position associated with the registered image feature selected in S19 is located within the identified range of several meters, it is determined that there is a match. On the other hand, if the imaging position associated with the registered image feature selected in S19 is not located within the identified range of several meters, it is determined that there is a match.

[0078] Then, if it is determined that the imaging position associated with the registered image features selected in S19 matches the current position of the vehicle identified in S20 based on sensor information (S21: YES), the process proceeds to S22. On the other hand, if it is determined that the imaging position associated with the registered image features selected in S19 does not match the current position of the vehicle identified in S20 based on sensor information (S21: NO), the process proceeds to S27.

[0079] In S22, the CPU 31 estimates that the vehicle is located at the imaging position corresponding to the registered image feature selected in S19. For example, as shown in Figure 10, if registered image feature quantities A to D are stored for imaging positions A to D on the parking track 72, and the current image feature quantity E acquired by the moving vehicle is similar to the registered image feature quantity B and also matches the current position of the vehicle identified by GPS, then the current position E of the vehicle is estimated to be imaging position B. Since the distance X, which is the interval at which registered image feature quantities are stored, is several centimeters to tens of centimeters as described above, the estimation accuracy is higher compared to estimating the current position of the vehicle using GPS.

[0080] Subsequently, in S23, the CPU 31 performs feature point extraction processing on the images captured most recently by the front camera 6, rear camera 7, and side cameras 8A and 8B. The details are the same as in S6, so the explanation will be omitted.

[0081] Subsequently, in S24, the CPU 31 reads the corresponding map information 57 (the map information generated in S6 when registering the parking trajectory and parking location that are the target of this parking assistance) from the generated map information DB 36, and matches the feature points 56 extracted in S23 with the point cloud of feature points 56 stored in the map information 57. Specifically, matching is performed by comparing the feature quantities of the feature points and associating feature points with high similarity. For example, methods such as brute-force matching and cross-checking are used. The point cloud of feature points to be matched is the point cloud of feature points stored in the map information within a first range from the vehicle's current position estimated in S24. By narrowing down the range of the point cloud of feature points to be matched in advance, the speed of determining the current position can be improved, and the current position can be determined more accurately and reliably.

[0082] Furthermore, the range of the point cloud of feature points to be matched in S24 is determined according to the accuracy of the vehicle's current position estimation in the most recent S22. Since S22 allows for highly accurate estimation of the vehicle's current position using image features, the first range is set to a relatively narrow range. For example, assuming that the vehicle is located within several tens of centimeters before or after the estimated current position, matching is performed on the point cloud of feature points that can be extracted from that range. On the other hand, as will be described later, if the accuracy of the vehicle's current position estimation is low, the range of the point cloud of feature points to be matched is set to a wider range than the first range (S29).

[0083] Subsequently, in S25, the CPU 31 determines the detailed current position and orientation (vehicle orientation) of the vehicle based on the matching results of S24. In this embodiment, Visual SLAM is used to determine the current position and orientation of the vehicle using map information 57 on which a point cloud of feature points 56 is arranged and feature points 56 extracted from the captured image in real time. However, LiDAR SLAM or Depth SLAM can also be used as methods for determining the position and orientation of the vehicle using feature points.

[0084] The CPU 31 then controls each drive unit 39 based on the identified vehicle's current position and attitude to provide automatic driving assistance for the vehicle 2. Specifically, the vehicle controls such as steering, drive source, and brakes are automatically performed so that the vehicle travels along a registered parking path at a specified speed. The shift position is also switched automatically.

[0085] However, when moving the vehicle along a parking path, only the steering operation may be automated, while the drive system and brakes may be controlled manually. Alternatively, the vehicle movement may be performed manually rather than automatically. In that case, as parking assistance, the parking space and parking path may be displayed on the LCD display 4, and the timing for turning the steering wheel may be announced by voice.

[0086] Furthermore, while parking assistance is being provided, it is desirable to display real-time bird's-eye and overhead images showing the current environment around the vehicle on the LCD display 4 as support images to assist the vehicle's autonomous driving. However, it is not necessary to display both the bird's-eye and overhead images simultaneously; the user may switch between them. In addition, if there are objects requiring attention around the vehicle, such as pedestrians or bicycles, it is desirable to display warning images of these objects in the bird's-eye or overhead images.

[0087] Furthermore, while vehicle control such as steering, drivetrain, and brakes is being performed automatically, the vehicle occupants can cancel the automatic driving assistance at any time of their own volition, and can also stop the vehicle by performing brake operations. When performing the above-mentioned automatic driving assistance, as described above, the LCD display 4 displays real-time bird's-eye and overhead images showing the environment around the vehicle, and the vehicle occupants can interrupt the automatic driving assistance and stop the vehicle by performing brake operations if necessary while viewing the display.

[0088] Next, in S26, the CPU 31 determines whether or not the vehicle has been parked. Specifically, the CPU determines that the vehicle has been parked when the vehicle is positioned in the registered parking location and the vehicle's shift position is changed to "P".

[0089] If it is determined that the vehicle has been parked (S26: YES), the parking assistance processing program is terminated. Conversely, if it is determined that the vehicle has not been parked (S26: NO), the parking assistance is continued.

[0090] On the other hand, in S21, if it is determined that the imaging position associated with the registered image feature quantity selected in S19 does not match the current position of the vehicle identified in S20 based on sensor information (S21: NO), S27 is executed, in which it is estimated that the vehicle is located at the position identified in S20 based on sensor information. However, the accuracy of GPS current position detection is not high, and the current position is estimated to be within a range of several meters, for example.

[0091] Subsequently, in S28, the CPU 31 performs feature point extraction processing on the images captured most recently by the front camera 6, rear camera 7, and side cameras 8A and 8B. The details are the same as in S6, so the explanation will be omitted.

[0092] Next, in S29, the CPU 31 reads the corresponding map information 57 (the map information generated in S6 when registering the parking trajectory and parking location that are the target of this parking assistance) from the generated map information DB 36, and matches the feature points 56 extracted in S28 with the point cloud of feature points 56 stored in the map information 57. Specifically, matching is performed by comparing the feature quantities of the feature points and associating feature points with high similarity. For example, methods such as brute-force matching and cross-checking are used. The point cloud of feature points to be matched is the point cloud of feature points stored in the map information within the second range from the current position of the vehicle estimated in S27.

[0093] Furthermore, the range of the point cloud of feature points to be matched in S29 is determined according to the accuracy of the vehicle's current position estimation in the most recent S27. Note that in S27, the vehicle's current position is estimated using GPS, so the accuracy of the vehicle's current position estimation is lower compared to S22. Therefore, the second range is set to be wider than the first range. In other words, the lower the accuracy of the vehicle's current position estimation, the wider the matching range is set. For example, assuming that the vehicle is located within a few meters before or after the estimated current position, matching is performed on the point cloud of feature points that can be extracted from that range. By doing so, even if the accuracy of the vehicle's current position estimation is low, the range of matching can be widened, preventing situations where the vehicle's current position cannot be determined.

[0094] Subsequently, in S30, the CPU 31 identifies the detailed current position and attitude (vehicle orientation) of the vehicle based on the matching results in S29. Then, based on the identified current position and attitude of the vehicle, it controls each drive unit 39 to provide automatic driving assistance for the vehicle 2. Specifically, the steering, drive source, brakes, and other vehicle controls are automatically performed so that the vehicle travels along a registered parking path at a specified speed. The shift position is also switched automatically. The details are the same as in S25. After that, the process proceeds to S26.

[0095] On the other hand, in S31, which is executed when it is determined in S15 that the processes below S15 have already been executed at least once since the start of long-range parking assistance, the CPU 31 determines whether the current state of the vehicle is in a state where the vehicle's current position is lost during the execution of parking assistance (hereinafter referred to as the lost state). The lost state occurs, for example, when the most recent captured image could not be acquired for some reason, or when, even if an captured image could be acquired, feature point matching could not be performed (for example, when a large obstacle such as a person or car is captured in a position that obstructs the camera's view).

[0096] If it is determined that the current state of the vehicle is lost (S31: YES), the process proceeds to S32. Conversely, if it is determined that the current state of the vehicle is not lost (S31: NO), the process proceeds to S33.

[0097] In S32, the CPU 31 acquires the detection results of sensors equipped on the vehicle 2 for determining the change in the vehicle's mileage and direction, specifically the vehicle speed sensor, steering sensor, and gyro sensor, as sensor information via CAN, and estimates the vehicle's current position by combining the acquired sensor information. More specifically, the current position of the vehicle can be estimated based on the vehicle position identified most recently before the vehicle was lost, and the change in the vehicle's mileage and direction from that position. Furthermore, the accuracy of detecting the current position using the above-mentioned vehicle speed sensor, steering sensor, and gyro sensor is higher than that of GPS, and the current position can be determined, for example, within a range of several tens of centimeters.

[0098] Subsequently, the process proceeds to S23, where the vehicle's current position is determined using feature points and the vehicle is controlled (S23-S25). The point cloud of feature points to be matched in S24 is the point cloud of feature points stored in the map information within the first range from the vehicle's current position estimated in S32. In S32, it is possible to estimate the vehicle's current position with high accuracy using vehicle speed pulses and changes in direction, so the first range is set to a relatively narrow range. For example, assuming that the vehicle is located within several tens of centimeters before or after the estimated vehicle's current position, matching is performed on the point cloud of feature points that can be extracted from that range.

[0099] Meanwhile, in S33, the CPU 31 performs feature point extraction processing on the images captured most recently by the front camera 6, rear camera 7, and side cameras 8A and 8B. The details are the same as in S6, so the explanation will be omitted.

[0100] Subsequently, in S34, the CPU 31 reads the corresponding map information 57 (the map information generated in S6 when registering the parking trajectory and parking position that are the target of this parking assistance) from the generated map information DB 36, and matches the feature points 56 extracted in S33 with the point cloud of feature points 56 stored in the map information 57. Specifically, matching is performed by comparing the feature quantities of the feature points and associating feature points with high similarity. For example, a method combining brute-force matching and cross-checking is used. The point cloud of feature points to be matched is narrowed down from the current position and orientation of the vehicle that has been identified at this time. That is, from the point cloud of feature points stored in the map information, the range of the point cloud of feature points to be matched next can be predicted from the current position and orientation of the vehicle that has been identified at this time, so matching is performed on the point cloud of feature points stored in the map information within the predicted range. By narrowing down the range of the point cloud of feature points to be matched in advance, the speed of identifying the current position is improved and the current position can be identified more accurately and reliably.

[0101] Subsequently, in S35, the CPU 31 identifies the new current position and attitude (vehicle orientation) of the vehicle based on the matching results of S34. Then, based on the identified current position and attitude of the vehicle, it controls each drive unit 39 to provide automatic driving assistance for the vehicle 2. Specifically, the steering, drive source, brakes, and other vehicle controls are automatically performed so that the vehicle travels along the registered parking path at a specified speed. The shift position is also switched automatically. The details are the same as in S25. After that, the process proceeds to S26.

[0102] As described in detail above, according to the driving support device 1 and the computer program executed by the driving support device 1 according to this embodiment, image features are extracted from images of the surrounding area captured at multiple different imaging positions by a camera equipped on the vehicle 2, and map information 57 is generated in which the image features extracted for each image are associated with the imaging position from which the source image was captured (S6). On the other hand, when the vehicle 2 is driving in the area where the map information 57 has been generated, image features extracted from images of the surrounding area captured by the camera during driving are acquired as current image features, and the detection result of the sensor 38 for identifying the current position of the vehicle 2 is acquired as sensor information (S20). The current position of the vehicle 2 is estimated using the current image features, sensor information and map information 57 acquired during driving in the area (S22, S27). After estimating the current position of vehicle 2, the system matches the estimated current position of vehicle 2 with a point cloud of feature points stored in map information 57 within a range corresponding to the accuracy of the estimation of vehicle 2's current position, and with feature points extracted from images of the surrounding area captured by a camera while driving through the area, thereby determining the vehicle's current position in more detail (S25, S30). This makes it possible to estimate the vehicle's current position to some extent before matching feature points, for example, at the initial timing of determining the vehicle's current position immediately after the start of support, or even if the vehicle's current position is lost for some reason. Furthermore, by narrowing down the feature points to be matched according to the accuracy of the estimation, it becomes possible to narrow down the range appropriately, enabling accurate and rapid identification of the vehicle's position. Furthermore, in estimating the current position of vehicle 2 (S22), the current image features are compared with registered image features stored in map information 57, and the registered image features closest to the current image features are selected (S19). It is then determined whether the imaging position associated with the selected registered image features matches the current position of the vehicle identified based on sensor information (S21). If it is determined that they match, the imaging position associated with the selected registered image features is estimated as the current position of vehicle 2. If it is determined that they do not match, the current position of vehicle 2 is estimated based on sensor information.As a result, when the vehicle's current position can be estimated using image features, it becomes possible to estimate the vehicle's current position with high accuracy using image features. On the other hand, even when the vehicle's current position cannot be estimated using image features, it is possible to estimate the approximate current position of the vehicle by using sensor information. Furthermore, the range corresponding to the accuracy of the estimation of the vehicle 2's current position is set to be narrower when the imaging position is estimated to be the vehicle 2's current position than when the vehicle's current position is estimated based on sensor information. As a result, when it is possible to estimate the vehicle's current position with high accuracy, the number of targets to be matched is reduced, enabling rapid identification of the vehicle's position. On the other hand, even when it is not possible to estimate the vehicle's current position with high accuracy, the number of targets to be matched is increased, thus avoiding situations where the vehicle's position cannot be determined at all. In addition, if there is a period during driving in the area when the vehicle's current position cannot be determined by matching, the vehicle's current position is estimated based on sensor information without using image features (S32). As a result, even if there is a period when the vehicle's current position cannot be determined by matching, it is possible to estimate the vehicle's current position by using sensor information.

[0103] It should be noted that the present invention is not limited to the embodiments described above, and various improvements and modifications are possible without departing from the spirit of the invention. For example, in this embodiment, as a method for determining the current position of a vehicle, particularly when performing long-range parking, the current position of the vehicle is estimated using image features, sensor information, and map information as described above, and then the current position of the vehicle is determined by Visual SLAM using the estimation results. However, the method for determining the current position of a vehicle of this application can be applied in any situation where it is necessary to determine the current position of a vehicle, and is not limited to parking assistance for long-range parking. For example, it can be applied while driving on a public road, while assisting with parking to a nearby parking position that is not long-range, or while driving manually.

[0104] Furthermore, in this embodiment, registered image features are stored in map information at predetermined distance X intervals (Figure 7), but if they are stored at different locations, they do not necessarily need to be stored at equal intervals.

[0105] Furthermore, in this embodiment, if the vehicle loses its current position during parking assistance, the current position of the vehicle is estimated by combining sensor information from sensors that determine the amount of change in distance traveled and direction, specifically a vehicle speed sensor, a steering sensor, and a gyro sensor (S32). However, the current position of the vehicle may also be estimated using GPS. In that case, however, the point cloud of feature points to be matched in the subsequent feature point matching will be the point cloud of feature points stored in the map information within a second range from the estimated current position of the vehicle.

[0106] Furthermore, in this embodiment, in the processing of S18 and S21 of the parking assistance processing program (Figure 8), it is checked whether the image features are consistent with the sensor information after comparison. However, the current position of the vehicle may be narrowed down first using the sensor information. Specifically, the CPU 31 first compares the vehicle position identified based on the sensor information with the imaging position stored in the map information 57, and selects an imaging position that is near the vehicle position identified based on the sensor information. For example, since the detection accuracy of the current position of GPS is to identify within a range of several meters, an imaging position within that range is selected. After that, it is determined whether the image features associated with the selected imaging position are consistent with the current image features. If it is determined that they are consistent, the selected imaging position is estimated to be the current position of the vehicle. On the other hand, if it is determined that they are not consistent, it is possible to perform the processing from S18 onwards.

[0107] Furthermore, the execution order of each step in the parking assistance information registration processing program shown in Figure 3 and the parking assistance processing program shown in Figure 8 is just an example, and the execution order can be changed as appropriate. For example, the image feature extraction process in S2 to S4 and the map information generation process in S6 may be executed after S8. In addition, the processing order of S16 to S19 and S20 may be reversed. It is also possible to omit some steps included in the parking assistance information registration processing program and the parking assistance processing program; for example, steps from S31 onwards may be omitted.

[0108] Furthermore, in this embodiment, the driver assistance ECU 10 of the driver assistance device 1 executes the parking assistance information registration processing program (Figure 3) and the parking assistance processing program (Figure 8), but the execution entity can be changed as appropriate. For example, the control unit of the liquid crystal display 4, the vehicle control ECU, the control unit of the navigation device, or other in-vehicle devices may be used to execute the processing. Alternatively, it is possible to execute the processing on a communication terminal that is connected to the vehicle 2 in a communicative manner. Moreover, when the processing is executed on a communication terminal, it is also possible to execute part of the processing on an external server.

[0109] [Summary of this embodiment] This embodiment comprises at least the following configurations. The vehicle (2) is equipped with imaging devices (6, 7, 8A, 8B) that capture images of the surrounding area at multiple different imaging positions, and image features are extracted from these images for each image. Map information (57) is generated by associating the image features extracted for each image with the imaging position from which the source image was captured. When the vehicle travels through the area where the map information is generated, the image features extracted from images of the surrounding area captured by the imaging devices during travel are acquired as current image features, and the detection results of a sensor (38) for identifying the vehicle's current position are acquired as sensor information. The current position of the vehicle is estimated using the current image features acquired during travel through the area, the sensor information, and the map information. After estimating the vehicle's current position, the vehicle's current position is determined in more detail by matching a point cloud of feature points stored in the map information within a range corresponding to the accuracy of the vehicle's current position estimation with the feature points extracted from images of the surrounding area captured by the imaging devices during travel through the area.

[0110] This configuration allows for the estimation of the vehicle's current location to some extent, even immediately after the start of support operations, or if the vehicle's location is lost for any reason, before feature point matching is performed. Furthermore, by narrowing down the feature points to be matched according to the accuracy of the estimation, it becomes possible to narrow down the range appropriately, enabling accurate and rapid identification of the vehicle's location.

[0111] Furthermore, in this embodiment, when estimating the current position of the vehicle (2), it is preferable to compare the current image feature quantity with the registered image feature quantity which is the image feature quantity stored in the map information (57), select the registered image feature quantity which is closest to the current image feature quantity, determine whether the imaging position associated with the selected registered image feature quantity is consistent with the current position of the vehicle which is identified based on the sensor information, and if it is determined that they are consistent, estimate the imaging position associated with the selected registered image feature quantity as the current position of the vehicle, while if it is determined that they are not consistent, estimate the current position of the vehicle based on the sensor information.

[0112] With this configuration, when the vehicle's current position can be estimated using image features, it is possible to estimate the vehicle's current position with high accuracy using image features. On the other hand, even when the vehicle's current position cannot be estimated using image features, it is possible to estimate the vehicle's approximate current position by using sensor information.

[0113] Furthermore, in this embodiment, it is preferable that the range corresponding to the accuracy of estimating the current position of the vehicle (2) is set to be narrower than the range when the imaging position is estimated to be the current position of the vehicle compared to when the current position of the vehicle is estimated based on the sensor information.

[0114] With this configuration, when it is possible to estimate the current location of a vehicle with high accuracy, the number of matching targets can be reduced to enable rapid identification of the vehicle's location. On the other hand, even when it is not possible to estimate the current location of a vehicle with high accuracy, the number of matching targets can be increased to avoid situations where the vehicle's location cannot be determined at all.

[0115] Furthermore, in this embodiment, if there is a period during driving in the area when the current position of the vehicle (2) cannot be determined by the matching, it is preferable to estimate the current position of the vehicle based on the sensor information without using the current image features.

[0116] With this configuration, even if there is a period when the vehicle's current location cannot be determined through matching, the vehicle's current location can be estimated using sensor information.

[0117] [Note] Furthermore, the embodiments described above also disclose the following inventions. In the following description, the names and expressions of corresponding components in the embodiments, and the reference numerals used in the drawings, are noted in parentheses for reference. However, the components of each invention are not limited to these notes.

[0118] (Invention A) The driving assistance device (1) according to claim 1, wherein the sensor information is information that identifies the absolute position of the vehicle (2), or information that identifies the amount of change in the vehicle's mileage and direction.

[0119] According to this, it becomes possible to estimate the vehicle's current position by using sensor information.

[0120] (Invention B) The driving support device (1) according to claim 1, wherein in estimating the current position of the vehicle (2), the device compares the position of the vehicle identified based on the sensor information with the imaging position stored in the map information (57), selects the imaging position located around the position of the vehicle identified based on the sensor information, determines whether the image feature amount associated with the selected imaging position matches the current image feature amount, estimates the selected imaging position as the current position of the vehicle if it is determined that they match, while estimates the selected imaging position as the current position of the vehicle if it is determined that they do not match, compares the current image feature amount with the registered image feature amount which is the image feature amount stored in the map information, selects the registered image feature amount which is closest to the current image feature amount, and estimates the imaging position associated with the selected registered image feature amount as the current position of the vehicle.

[0121] According to this, it becomes possible to determine the vehicle's current location, for example, at the very beginning of support operations, or even if the vehicle's current location is lost for some reason, and to estimate its current location. Furthermore, by narrowing down the feature points to be matched according to the accuracy of the estimation, it becomes possible to narrow down the search to an appropriate range, enabling accurate and rapid identification of the vehicle's location.

[0122] (Invention C) The driving support device (1) according to Invention A, wherein the sensor information is information that identifies the amount of change in the travel distance and direction of the vehicle (2), and if a period occurs during driving in the area during which the current position of the vehicle cannot be determined by the matching, the current position of the vehicle is estimated using the most recently determined current position of the vehicle and the sensor information.

[0123] According to this, even if the vehicle's current location is lost for some reason, it becomes possible to accurately estimate the vehicle's current location using sensor information.

[0124] 1...Driving assistance system, 2...Vehicle, 3...Control unit, 4...LCD display, 6...Front camera (imaging device), 7...Rear camera (imaging device), 8A, 8B...Side cameras (imaging devices), 10...Driving assistance ECU, 31...CPU, 38...Sensor, 55...Captured image, 56...Feature points, 57...Map information

Claims

1. A driving assistance device that extracts image features from images of the surrounding area captured at multiple different imaging positions by an imaging device installed in the vehicle, generates map information by associating the extracted image features with the imaging position from which the source image was captured, and when the vehicle is traveling in the area where the map information is generated, acquires the image features extracted from images of the surrounding area captured by the imaging device during the journey as current image features, and acquires the detection results of a sensor for identifying the vehicle's current position as sensor information, estimates the vehicle's current position using the current image features, sensor information and map information acquired during the journey in the area, and then identifies the vehicle's current position in more detail by matching a point cloud of feature points stored in the map information within a range corresponding to the accuracy of the estimation of the vehicle's current position with feature points extracted from images of the surrounding area captured by the imaging device during the journey in the area.

2. In estimating the current position of the vehicle, the driver assistance device according to claim 1 compares the current image feature quantity with the registered image feature quantity which is the image feature quantity stored in the map information, selects the registered image feature quantity which is closest to the current image feature quantity, determines whether the imaging position associated with the selected registered image feature quantity is consistent with the current position of the vehicle which is determined based on the sensor information, and if it is determined that they are consistent, estimates the imaging position associated with the selected registered image feature quantity as the current position of the vehicle, while if it is determined that they are not consistent, estimates the current position of the vehicle based on the sensor information.

3. The driving assistance device according to claim 2, wherein the range corresponding to the accuracy of estimating the current position of the vehicle is set to be narrower than the range when the current position of the vehicle is estimated based on the sensor information when the imaging position is estimated to be the current position of the vehicle.

4. If a period occurs during driving in the area during which the current position of the vehicle cannot be determined by matching, the driving support device according to claim 1, which estimates the current position of the vehicle based on the sensor information without using the current image features.

Images