Moving body control device, moving body control method, and program

Abstract

A moving body control device according to an embodiment of the present invention comprises: an acquisition unit that acquires an image which includes a road surface around a moving body; and a recognition unit that extracts contour lines of the travel path boundary of the moving body from the image, that distinguishes and recognizes, from the extracted contour lines, a first direction contour line which extends in a first direction and a second direction contour line which extends in a second direction different from the first direction, and that recognizes the shape of a road around the moving body on the basis of the plurality of recognized contour lines.

Description

Mobile body control device, mobile body control method, and program

[0001] The present invention relates to a mobile body control device, a mobile body control method, and a program.

[0002] In recent years, efforts to provide access to sustainable transportation systems that also take into consideration vulnerable people among traffic participants have become active. To achieve this, efforts are focused on research and development to further improve traffic safety and convenience through research and development related to autonomous driving technology. In relation to this, conventionally, at an intersection where a travel route turning from the own lane to another lane is set, a first extension line extending through the position of another vehicle moving from the direction opposite to the turning direction of the own vehicle toward the turning direction in the other lane, extending in the moving direction of the other vehicle or the opposite direction; an outermost boundary line which is an arc-shaped curve having a radius equal to the minimum turning radius of the own vehicle tangent to a first axle direction line extending in the axle direction of the own vehicle through the position of the own vehicle and the first extension line; and an innermost boundary line which is an arc-shaped curve having a radius larger than the minimum turning radius are set. A technology is known that sets the turning path of the own vehicle at the intersection in a path setting area that is an area surrounded by the first extension line, the first axle direction line, the outermost boundary line, and the innermost boundary line (see, for example, Patent Document 1).

[0003] Japanese Unexamined Patent Publication No. 2022-83093

[0004] By the way, in autonomous driving technology, when recognizing road shapes such as intersections, there have been cases where road shapes such as intersecting roads and branching roads cannot be accurately recognized only by the output result of an external environment detection device such as a camera mounted on the mobile body. Therefore, conventionally, there has been a problem that it may not be possible to appropriately recognize the traveling path of the mobile body and appropriately control the mobile body.

[0005] In order to solve the above problem, an object of the present application is to provide a mobile body control device, a mobile body control method, and a program that can more appropriately recognize the traveling path of a mobile body and control the mobile body more appropriately. And by extension, it contributes to the development of a sustainable transportation system.

[0006] The mobile body control device, mobile body control method, and program according to this invention employ the following configuration: (1) A mobile body control device according to one aspect of this invention is a mobile body control device comprising: an acquisition unit that acquires an image including the road surface around a mobile body; and a recognition unit that extracts the contour lines of the road boundary of the mobile body from the image, distinguishes and recognizes a first direction contour line extending in a first direction from the extracted contour line and a second direction contour line extending in a second direction different from the first direction, and recognizes the road shape around the mobile body based on the recognized plurality of contour lines.

[0007] (2) In the embodiment of (1) above, when the recognition unit recognizes a pair of first directional contour lines or a pair of second directional contour lines, it searches for the center point of the two recognized contour lines and recognizes the road shape based on the center point that was searched.

[0008] (3) In the embodiment of (2) above, the recognition unit recognizes that the location where the center point search is interrupted is an area where deceleration control of the moving body is required, when the continuous center point search is interrupted and a second directional contour line extending in a direction different from the first directional contour line is recognized at the location where the search is interrupted.

[0009] (4) In the embodiment of (2) above, when the recognition unit recognizes a pair of first directional contour lines or a pair of second directional contour lines, it searches for the center point of the two recognized contour lines, and recognizes that they are the center points of the two contour lines if the number of center points at a predetermined distance is equal to or greater than a predetermined number.

[0010] (5) In the embodiment of (1) above, the recognition unit recognizes that a road formed by a pair of contour lines is a Type 1 road when the distance between the center of a pair of contour lines and the contour line is greater than or equal to a threshold, and recognizes that a road formed by a pair of contour lines is a Type 2 road when the distance is less than the threshold.

[0011] (6) In the embodiment of (5) above, when the recognition unit recognizes a plurality of first-class roads, the system further includes a display control unit that outputs each of the plurality of first-class roads to the display unit in a different display manner.

[0012] (7) In the embodiment of (5) above, if the recognition unit recognizes multiple Type 1 roads, the system further includes an inquiry unit that inquires with the occupant of the mobile body which of the multiple Type 1 roads will be used as the route for the mobile body.

[0013] (8) In the embodiment of (5) above, when the recognition unit recognizes a plurality of the first type roads, the vehicle further comprises a driving control unit that moves the vehicle to the first type road selected by an instruction from the occupant operating the vehicle.

[0014] (9): In any one of the embodiments of (1) to (8) above, when the recognition unit recognizes at least three contour lines extending in the same direction, it searches for the center points of adjacent contour lines among the at least three contour lines, determines a pair of contour lines based on the positions of the searched center points, and recognizes the road shape around the moving body based on the positions of the determined pair of contour lines.

[0015] (10): In any one embodiment of (1) to (8) above, the recognition unit integrates the contour lines of the road boundary extracted from each of the plurality of images based on position information when the acquisition unit acquires a plurality of images of the moving object in different directions.

[0016] (11): In any one of the embodiments of (1) to (8) above, when the acquisition unit acquires multiple images captured at different times, the recognition unit compares the contour lines acquired from each image, and if the distance between the contour lines is less than a threshold and the extension direction of each contour line is within a predetermined range, the recognition unit determines that the compared contour line segments are the same contour and integrates them.

[0017] (12): A mobile body control device according to another aspect of the present invention is a mobile body control device comprising: an acquisition unit that acquires an image including the road surface around a mobile body; a lane recognition unit that extracts the contour lines of the road boundary of the mobile body from the image, searches for the center points of pairs of contour line segments among the extracted plurality of contour lines, performs clustering based on the positions of the searched center point cloud, and recognizes the position and shape of a partial lane based on the position information of the clustered center point cloud; and a recognition unit that recognizes the road shape based on the position and shape of the lane recognized by the lane recognition unit.

[0018] (13): A mobile body control method according to another aspect of the present invention is a mobile body control method in which a computer acquires an image including the road surface around the mobile body, extracts contour lines of the road boundary of the mobile body from the image, distinguishes and recognizes first direction contour lines extending in a first direction from the extracted contour lines and second direction contour lines extending in a second direction different from the first direction, and recognizes the road shape around the mobile body based on the recognized plurality of contour lines.

[0019] (14): A program according to another aspect of the present invention is a program that causes a computer to acquire an image including the road surface around a moving object, to extract contour lines of the road boundary of the moving object from the image, to distinguish and recognize first direction contour lines extending in a first direction from the extracted contour lines and second direction contour lines extending in a second direction different from the first direction, and to recognize the road shape around the moving object based on the recognized plurality of contour lines.

[0020] According to the embodiments described in (1) to (14) above, the path of the moving object can be recognized more effectively and the moving object can be controlled more appropriately.

[0021] This figure shows an example of the configuration of the mobile body 1 and control device 100 according to the embodiment. This is a perspective view of the mobile body 1 seen from above. This figure shows an example of the functional configuration of the recognition unit 140. This figure illustrates an example of the function performed by the extraction unit 141. This figure illustrates how to obtain the position of contour points from the mask image IM10. This figure illustrates how to set the order of contour points relative to the track boundary TB. This figure shows an example of the image IM20 after homography transformation. This figure illustrates how to aggregate the track boundary TB. This figure illustrates a specific example of contour line integration based on the spatial axis. This figure illustrates a specific example of contour line integration based on the time axis. This figure illustrates the details of the function of the lane recognition unit 142. This figure illustrates how to search for a center point. This figure illustrates how to search for a pair of two contour line segments. This figure illustrates how to search for a center point. This figure illustrates how to cluster center points. This figure shows the state before and after the process of aligning the orientation of center points. This figure illustrates how to cluster contour line segments. This figure illustrates the details of the function of the connection unit 143. This figure illustrates how to enumerate connection candidates. This figure illustrates how to find connection line segments. This is a diagram illustrating the overlap between candidate connection lines and other lanes. This is a diagram illustrating the optimization of connection relationships. This is a diagram showing an example of an image displayed in the embodiment. This is a diagram showing another example of an image displayed in the embodiment. This is a flowchart showing an example of the processing flow executed by the control device 100 of the embodiment.

[0022] The following describes embodiments of the mobile vehicle control device, mobile vehicle control method, and program of the present invention with reference to the drawings. In the following description, the mobile vehicle control device is assumed to be mounted on the mobile vehicle. The mobile vehicle is capable of moving on both roadways and predetermined areas separate from roadways. The mobile vehicle is sometimes referred to as a micromobility. An electric kick scooter is a type of micromobility. The predetermined area is, for example, a sidewalk. Alternatively, the predetermined area may be part or all of a road shoulder, bicycle lane, or public open space. The mobile vehicle may also have driving assistance functions such as automatic driving. Automatic driving refers to, for example, automatically controlling the steering or speed of the mobile vehicle, or both, to perform driving control. The above-mentioned driving control may include, for example, LKAS (Lane Keeping Assistance System), ACC (Adaptive Cruise Control System), ALC (Automated Lane Change), CMBS (Collision Mitigation Brake System), and other driving control systems. Furthermore, the mobile vehicle may also be controlled manually by the occupant of the mobile vehicle (e.g., the driver) (so-called manual driving).

[0023] Figure 1 shows an example of the configuration of a mobile body 1 and a control device 100 according to an embodiment. The mobile body 1 is equipped with, for example, an external environment detection device 10, a mobile body sensor 12, an operator 14, an internal camera 16, a positioning device 18, a communication device 20, an HMI (Human Machine Interface) 30, a drive device 40, a moving mechanism 50, a storage device 70, and a control device 100. Some of these components that are not essential for realizing the functions of the present invention may be omitted.

[0024] The external environment detection device 10 is a variety of devices whose detection range is at least the direction of movement of the moving object 1. The external environment detection device 10 detects the external conditions of the moving object 1. The external environment detection device 10 includes, for example, an external camera 11. The external camera 11 is an example of an "imaging unit". In addition to the external camera 11, the external environment detection device 10 may also include a radar device, LIDAR (Light Detection and Ranging), etc.

[0025] The external camera 11 is a digital camera that utilizes a solid-state image sensor such as a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor). The external camera 11 is attached to any point on the moving body 1 that can capture the surrounding area including the direction of movement of the moving body 1. Multiple external cameras 11 may be provided to capture the sides and rear of the moving body 1. When multiple external cameras 11 are provided, each captured image (RGB image) can be synchronized using time information or the like. The external camera 11 captures the surrounding area of ​​the moving body 1 periodically and repeatedly. The external camera 11 may also be a stereo camera.

[0026] The radar device emits radio waves such as millimeter waves around the moving object 1 and detects the radio waves reflected by the object (reflected waves) to detect at least the object's position (distance and direction). The radar device can be attached to any location on the moving object 1. The radar device may also detect the object's position and velocity using the FM-CW (Frequency Modulated Continuous Wave) method. The LIDAR irradiates the moving object 1 with light (or electromagnetic waves with a wavelength close to light) and measures the scattered light. The LIDAR detects the distance to the target based on the time from emission to reception. The irradiated light is, for example, pulsed laser light. The LIDAR can be attached to any location on the moving object 1. The external detection device 10 outputs information (image, etc.) indicating the detection result to the control device 100.

[0027] The moving body sensor 12 includes, for example, a speed sensor for detecting the speed of the moving body 1, an acceleration sensor for detecting acceleration, a yaw rate (angular velocity) sensor for detecting the yaw rate (for example, the rotational angular velocity around the vertical axis passing through the center of gravity of the moving body 1), a compass sensor, and an operator sensor attached to the operator 14.

[0028] The control element 14 receives driving operations from the occupant of the mobile vehicle 1. The control element 14 includes, for example, controls for instructing acceleration and deceleration (e.g., an accelerator pedal, a brake pedal, a dial switch or lever for speed adjustment) and controls for instructing steering (e.g., a steering wheel). In this case, the mobile vehicle sensor 12 may include controls for detecting the amount of operation, such as an accelerator opening sensor, a brake pedal pressure sensor, and a steering torque sensor. The mobile vehicle 1 may also be equipped with controls other than those described above as the control element 14 (e.g., a non-annular rotary control, a joystick, a button, etc.).

[0029] The internal camera 16 captures images from the front of at least the head of the occupant of the mobile vehicle 1 (for example, the driver). The internal camera 16 is a digital camera that uses an image sensor such as a CCD or CMOS. The internal camera 16 outputs the captured image to the control device 100.

[0030] The positioning device 18 is a device that determines the position of the mobile body 1. The positioning device 18 is, for example, a GNSS (Global Navigation Satellite System) receiver, which determines the position of the mobile body 1 based on signals received from GNSS satellites and outputs it as position information. The position information of the mobile body 1 may be estimated from the position of the Wi-Fi base station to which the communication device 20 mounted on the mobile body 1 is connected.

[0031] The communication device 20 communicates with other mobile devices in the vicinity of the mobile device 1, or communicates with various server devices via a wireless base station, using networks such as a cellular network, Wi-Fi network, Bluetooth®, or DSRC (Dedicated Short Range Communication).

[0032] The HMI 30 presents various information to the occupant of the mobile unit 1 and accepts input operations from the occupant. The HMI 30 includes, for example, a display unit and a speaker. The display unit is, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) display device. The display unit displays various images (including video) in the embodiment. The display unit may be configured as a touch panel and integrated with the input unit. The speaker outputs predetermined sounds (for example, alarms).

[0033] Furthermore, the HMI 30 may include a microphone, buzzer, touch panel, keys, etc. Also, the HMI 30 may include external notification devices such as lamps, display devices, and speakers that are provided on the outer surface of the mobile body 1 and notify information to the outside of the mobile body 1.

[0034] The drive unit 40 outputs a driving force (torque) to the moving mechanism 50 for the mobile body 1 to move. For example, the drive unit 40 includes a motor that drives the drive wheels, a battery that stores the power supplied to the motor, a steering device that adjusts the steering angle of the steering wheels, etc. The drive unit 40 may also be equipped with an internal combustion engine or a fuel cell as a means of outputting driving force or a means of generating power. The drive unit 40 may also be further equipped with a braking device that uses friction or air resistance. The drive unit 40 receives instructions from the operator 14 or the driving control unit 162 and causes the moving mechanism 50 to perform an operation according to the received instructions.

[0035] The mobility mechanism 50 is a mechanism for moving the mobile body 1 on a road (runway) or the like. The mobility mechanism 50 is, for example, a group of wheels including steering wheels and drive wheels. Alternatively, the mobility mechanism 50 may be legs for multi-legged walking.

[0036] Figure 2 is a perspective view of the mobile body 1 from above. In the figure, FW is the steering wheel, RW is the drive wheel, SD is the steering mechanism, MT is the motor, and BT is the battery. The steering mechanism SD, motor MT, and battery BT are included in the drive unit 40. Also in the figure, AP is the accelerator pedal, BP is the brake pedal, WH is the steering wheel, DI is the display unit, SP is the speaker, MC is the microphone, and EN is an external notification device located near the front end of the mobile body 1. The accelerator pedal AP, brake pedal BP, and steering wheel WH are included in the control unit 14. The display unit DI, speaker SP, microphone MC, and external notification device EN are included in the HMI 30.

[0037] In the illustrated example, the mobile vehicle 1 is a single-seater four-wheeled vehicle, and the occupant (driver) D of the mobile vehicle 1 is seated in the driver's seat DS and wearing a seat belt SB. Arrow A1 indicates the direction of travel (velocity vector) of the mobile vehicle 1. The external environment detection device 10 is provided near the front end of the mobile vehicle 1, and the internal camera 16 is positioned to capture images of the occupant D's head (face) from in front of the occupant D. In this embodiment, the mobile vehicle 1 is not limited to a single-seater, but may be capable of carrying multiple people.

[0038] Returning to Figure 1, the storage device 70 is, for example, a non-transient storage device such as an HDD (Hard Disk Drive), flash memory, or RAM (Random Access Memory). The storage device 70 stores map information 72, a program 74 executed by the control device 100, etc. In the figure, the storage device 70 is shown outside the frame of the control device 100, but the storage device 70 may be included in the control device 100. The map information 72 is, for example, information in which the shape of a road is represented by links indicating roads and nodes connected by links. The map information 72 may also include POI (Point of Interest) information, etc. Furthermore, the map information 72 may also include lane boundary information such as road markings (hereinafter referred to as markings) that demarcate lanes. Furthermore, the map information 72 may also include road information such as curvature (or radius of curvature), gradient, width (road width) of roads (or each lane included in a road), traffic regulation information, address information (address and postal code), facility information, telephone number information, etc. The map information 72 may be updated as needed by communicating with other devices via a communication device 20 or the like mounted on the mobile unit 1. In this embodiment, the map information 72 may not be present.

[0039] [Control device] The control device 100 includes, for example, an acquisition unit 120, a recognition unit 140, and a control unit 160. These components are realized, for example, by a hardware processor such as a CPU (Central Processing Unit) executing a program (software) 74. Some or all of these components may be realized by hardware (including circuitry) such as an LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), or GPU (Graphics Processing Unit), or by the cooperation of software and hardware. The program may be stored in the storage device 70 in advance, or it may be stored on a removable storage medium (non-transient storage medium) such as a DVD or CD-ROM and installed in the storage device 70 when the storage medium is mounted on a drive device. The acquisition unit 120, the recognition unit 140, and the control unit 160 are an example of a "mobile device control device".

[0040] The acquisition unit 120 acquires information from each component mounted on the mobile body 1 other than the control device 100 (for example, the external environment detection device 10, the mobile body sensor 12, the internal camera 16, the positioning device 18, the communication device 20, the HMI 30, the storage device 70, etc.). For example, the acquisition unit 120 acquires an image of the surroundings of the mobile body 1 captured by the external camera 11 (hereinafter referred to as the "external camera image").

[0041] Furthermore, the acquisition unit 120 may generate a mask image by performing image analysis on the external camera image (RGB image) using, for example, a DNN (Deep Neural Network) model generated by the function of deep learning, which is one of the machine learning functions of AI (Artificial Intelligence). This mask image divides the area within the image into a movable area where the mobile object 1 can move (e.g., the road area) and an immovable area where the mobile object 1 cannot move (e.g., the off-road area). Known techniques such as semantic segmentation can be used for image analysis. In addition, the model is not limited to DNN, and various models such as FCN (Fully Convolutional Network), CNN (Convolutional Neural Network), R-CNN (Region Convolutional Neural Network), FPN (Feature Pyramid Networks), and RNN (Recurrent Neural Network) may be used. These models may be stored in the storage device 70 in advance, or they may be acquired from an external source via the communication device 20.

[0042] For example, the acquisition unit 120 classifies each pixel in the image frame into a class (e.g., roadway, sidewalk, curb, obstacle, etc.) and assigns a label to it. It recognizes the pixel portion labeled as a roadway as a movable area and the pixel portion labeled as a sidewalk, curb, or obstacle as an immovable area. The acquisition unit 120 may also recognize the sidewalk as a movable area. Furthermore, the acquisition unit 120 may recognize the curb area as a boundary area between the movable area and the immovable area. The boundary area is an area included in the immovable area.

[0043] Furthermore, instead of (or in addition to) the above-described method, the acquisition unit 120 may perform known image analysis processing (for example, edge extraction processing, feature quantity extraction, pattern matching processing, etc.) on the external camera image, recognize other moving objects, pedestrians, etc. included in the external camera image based on the analysis result, further recognize an area where other moving objects are moving as a roadway, and recognize an area where pedestrians are moving as a sidewalk. The acquisition unit 120 recognizes a movable area and an immovable area based on the above-described recognition results of the roadway area and the sidewalk area.

[0044] The recognition unit 140 recognizes the surrounding conditions of the moving object 1 and the like based on the information acquired by the acquisition unit 120. Details of the functions of the recognition unit 140 will be described later.

[0045] The control unit 160 controls the entire respective components included in the moving object 1. The control unit 160 includes, for example, a travel control unit 162, an HMI control unit 164, and an inquiry unit 166. The HMI control unit 164 is an example of a "display control unit".

[0046] The driving control unit 162 controls the drive unit 40, etc., based on, for example, the recognition results from the recognition unit 140, and performs driving control that controls at least one of the steering or speed of the mobile body 1. For example, the driving control unit 162 performs steering control (LKAS control) so that the mobile body 1 travels in the center of the road recognized by the recognition unit 140 (or so that the mobile body 1 does not deviate from the lane markings (boundary lines) that demarcate the road). In addition, if, for example, the driving control unit 162 recognizes that the driver's state recognized by the recognition unit 140 is not suitable for driving the mobile body 1 (for example, sleeping or distracted), it may perform control to stop the mobile body 1 in a safe position such as the shoulder of the road. Furthermore, the driving control unit 162 may perform the above-mentioned driving control to avoid contact between the mobile body 1 and an obstacle recognized by the recognition unit 140. Furthermore, the driving control unit 162 may control at least one of the steering or speed of the mobile body 1 in response to instructions from the occupant D input from the HMI 30, and perform driving control such as ACC or ALC. Alternatively, the driving control unit 162 may refer to the map information 72 to determine a route from the current position to the destination set by the occupant D, and perform driving control so that the mobile body 1 moves along the determined route.

[0047] The HMI control unit 164 notifies a crew member D of the moving body 1 of predetermined information via the HMI 30, and accepts information input via the HMI 30. The predetermined information includes, for example, information related to the traveling of the moving body 1, such as information about the state of the moving body 1 (e.g., speed, current position, remaining fuel, etc.) and information about travel control. The information about travel control includes, for example, information about whether travel control is executed by the travel control unit 162 and information about the execution status thereof. Furthermore, the predetermined information may include information about surrounding conditions recognized by the external environment detection device 10. In addition, the predetermined information may include information unrelated to the traveling of the moving body 1, such as television programs and content (e.g., movies) stored in a storage medium such as a DVD. Furthermore, the HMI control unit 164 may cause the HMI 30 to output inquiry information to the crew member D, the recognition result obtained by the recognition unit 140, and the like. In addition, the HMI control unit 164 may cause the HMI 30 to output an alarm, for example, when there is a possibility that the moving body 1 collides with an obstacle or the like based on the relative position, relative speed, or the like between the moving body 1 and the obstacle.

[0048] The inquiry unit 166 makes an inquiry to the crew member D of the moving body 1 about a route that the moving body 1 should take, based on the recognition result of the recognition unit 140. Details of the function of the inquiry unit 166 will be described later.

[0049] [Recognition Unit] Next, details of the functions of the recognition unit 140 will be described. Fig. 3 is a diagram showing an example of the functional configuration of the recognition unit 140. The recognition unit 140 includes, for example, an extraction unit 141, a lane recognition unit 142, a connection unit 143, a road shape recognition unit 144, and an object recognition unit 145. Details of each configuration will be specifically described below.

[0050] [Extraction Unit] Figure 4 is a diagram illustrating an example of the functions performed by the extraction unit 141. For example, the extraction unit 141 obtains the positions of points (contour points) on the contour line of the path boundary in a mobile coordinate system with the reference position of the mobile body 1 (e.g., center of gravity or center) as the origin, based on the mask image obtained by the acquisition unit 120. Alternatively, the extraction unit 141 may obtain an external camera image to generate a mask image divided into movable and immovable areas, and then obtain the positions of the contour points of the path boundary in the mobile coordinate system.

[0051] Figure 5 illustrates how to obtain the positions of contour points from a mask image IM10. In the example in Figure 5, a mask image IM10 is shown that includes a movable area AR10 and an immovable area AR20. For example, the extraction unit 141 extracts the contour line, which is the outer frame of the boundary (path boundary TB) between the movable area AR10 and the immovable area AR20 shown in the mask image IM10, and further extracts the positions (pixels) of multiple contour points P from the contour line. For example, if the contour line is a straight line on the image, the extraction unit 141 may extract contour points P at equal intervals along the contour line, or if there is a change in the shape of the contour line, it may extract change points (corners, ends, inflection points, etc.) as contour points P. This extraction results in a larger number of contour points P in areas with complex shapes. Alternatively, the extraction unit 141 may perform a known contour detection process using, for example, the Teh-Chin chain approximation algorithm, instead of (or in addition to) the method described above, and extract the position of the contour point P from the contour of the movable area AR 10 obtained as a result of that process.

[0052] Next, the extraction unit 141 sets the order of the multiple contour points P, which are connecting elements of the contour line of a single track boundary TB, based on the positional relationship of the multiple contour points P. Figure 6 is a diagram illustrating how to set the order of contour points for a track boundary TB. For example, the extraction unit 141 sets the order of the contour points P extracted for a single track boundary TB, for example, in a clockwise direction (in the direction of the arrow in the figure). Alternatively, the extraction unit 141 may set the order of the contour points P in a counterclockwise direction.

[0053] Returning to Figure 4, the extraction unit 141 then performs a coordinate transformation (homography transformation) from the coordinate system (image coordinate system) of the mask image (external camera image) to the mobile body coordinate system, which is a top-down view of the mobile body 1, with the reference position of the mobile body 1 (for example, the centroid or center) as the origin (0,0). Figure 7 shows an example of the image IM20 after the homography transformation. In the example in Figure 7, the image IM20 is shown after projecting the image coordinate system to the mobile body coordinate system. The image IM20 shows four track boundaries TB1 to TB4 and contour points P corresponding to the boundary lines of each track boundary. For the sake of explanation, the contour points P extracted for track boundary TB1 are identified and shown as contour points Pa to Ph.

[0054] The extraction unit 141 transforms the mask image from which the contour points P have been extracted into a moving body coordinate system as shown in Figure 7. The order of the multiple contour points P for each of the track boundaries TB1 to TB4 remains unchanged after the homography transformation, and they are arranged in a clockwise (or counterclockwise) order relative to each connected element. For example, for the contour line of track boundary TB1, the order of contour points Pa→Pb→Pc→Pd→Pe→Pf→Pg→Ph is set in a clockwise direction.

[0055] Returning to Figure 4, the extraction unit 141 then performs a simplification process for the contour lines of the track boundary TB. For example, the extraction unit 141 applies the Douglas-Peucker algorithm or the Visvalingam-Whyatt algorithm to each track boundary TB to delete unnecessary contour points P. For example, the Douglas-Peucker algorithm draws a straight line to two contour points that are at a predetermined distance from each other (for example, contour points at both ends of the track boundary TB), and deletes contour points P whose distance from the straight line to other contour points is shorter than a threshold. For example, the Visvalingam-Whyatt algorithm deletes contour points P whose area is less than or equal to a threshold when a triangle is formed by three consecutive points on the track boundary TB is formed. In other words, the extraction unit 141 deletes contour points P that are not at the ends of the track boundary TB, among a plurality of contour points P corresponding to the contour lines of the track boundary TB, and which are predicted to cause little change in the contour shape of the track boundary TB even if deleted. After performing those processes, the contour of the track boundary TB can be generated using the remaining contour points P, thereby simplifying the track boundary TB while maintaining its general shape.

[0056] For example, for the road boundary TB1 shown in Figure 7, the extraction unit 141 deletes all contour points Pa to Ph on the contour line of the road boundary TB1 except for contour points Pa, Pb, Pe, and Pf that are located at the ends of the road boundary TB1 (contour points Pc, Pd, Pg, and Ph). A similar process is performed for road boundaries TB2 to TB4. As a result, the number of contour points P on the contour line is reduced, which reduces the processing load in subsequent processes (for example, center point search and enumeration of connection candidates, as described later). Therefore, the recognition of road shapes can be accelerated.

[0057] Returning to Figure 4, the extraction unit 141 then performs time synchronization on each external camera image when the multiple external cameras 11 mounted on the mobile body 1 capture images in multiple directions relative to the mobile body 1 (for example, forward, left side, right side, and rear), and integrates (combines, concatenates) the multiple external camera images to aggregate the track boundary TB contained in each camera image.

[0058] Figure 8 is a diagram illustrating the aggregation of track boundary TBs. The example in Figure 8 shows an example of integrating images captured by multiple cameras (front camera, left camera, right camera, and rear camera). In the example in Figure 8, contour points P extracted from the image captured by the front camera (external camera image) are indicated by square marks, contour points P extracted from the image captured by the left camera are indicated by star marks, contour points P extracted from the image captured by the right camera are indicated by circles, and contour points P extracted from the image captured by the rear camera are indicated by triangle marks.

[0059] When images of the surroundings of the moving object 1 are acquired using multiple cameras, the extraction unit 141 performs time synchronization based on time information contained in the external camera images and concatenates the images captured at the same time based on position information and imaging direction (field of view information of the external camera 11, etc.). Furthermore, the extraction unit 141 aggregates the track boundaries contained in each image. In the example in Figure 8, the aggregated result shows contour points P recognized from the images of the four cameras described above for the track boundaries TB1 to TB5 around the moving object 1. Note that time synchronization processing is not performed when only one camera is used for shooting.

[0060] Returning to Figure 4, the extraction unit 141 then integrates the contour lines of the track boundary TB. For example, the extraction unit 141 performs contour line integration using at least one of the following: contour line integration based on the spatial axis and contour line integration based on the time axis.

[0061] [Contour Integration Based on Spatial Axis] Figure 9 is a diagram illustrating a specific example of contour integration based on the spatial axis. In the example in Figure 9, the contour lines before integration (Figure 9-A) and after integration (Figure 9-B) are shown. Before integration (Figure 9-A), contour point P-1 (marked with a circle in the figure) on the contour line of the track boundary TB5-1 extracted from the external camera image captured by the right-side camera, and contour point P-2 (marked with a triangle in the figure) on the contour line of the track boundary TB5-2 extracted from the external camera image captured by the rear-side camera are shown.

[0062] For example, when contour lines observed simultaneously by multiple cameras are aggregated, if the fields of view (imaging range) of each camera overlap, there is a high probability that the contour lines will also overlap. Therefore, the extraction unit 141 integrates overlapping contour lines based on the spatial axis, as shown in Figure 9-A. Specifically, for each of the contour points P-1 and P-2 extracted from images captured by different cameras, the extraction unit 141 extracts contour line segments by connecting the contour points with straight lines. Then, the extraction unit 141 compares the contour line segments in the moving coordinate system, deletes one of the contour line segments whose distance is less than a threshold and whose extension direction is within a predetermined range, and connects and integrates the remaining contour line segments. The extension direction being within a predetermined range means, for example, that the angle (deviation angle) formed by the contour line segment connecting the points of contour point P-1 with a straight line and the contour line segment connecting the points of contour point P-2 with a straight line is less than a predetermined angle. As a result, after integration, the contour line of the road boundary has a reduced number of contour points P, as shown in Figure 9-B, thus simplifying and speeding up subsequent processing. Furthermore, even if the integrated contour points P are extracted from multiple external camera images, they may be displayed with the same mark, as shown in Figure 9-B. This reduces the processing load during image generation.

[0063] [Contour Integration Based on Time Axis] Figure 10 is a diagram illustrating a specific example of contour integration based on the time axis. In contour integration based on the time axis, the extraction unit 141 integrates contours extracted from images captured over multiple time steps (predetermined periods) by the same external camera 11 in order to stabilize the recognition results. For example, contour segments obtained by connecting contour points extracted from past external camera images with straight lines are compared with contour segments obtained by connecting contour points with straight lines from external camera images captured a predetermined time after the previous external camera image (for example, external camera images several frames later). Contour segments whose distance is less than a threshold and whose extension direction is within a predetermined range are determined to be the same contour and are connected.

[0064] In the example in Figure 10, the outlines (contour shapes) C0 to C4 of the track boundary extracted from an external camera image captured at time t(k-1), one frame prior to the current time t(k), and the outlines C5 to C9 of the track boundary extracted from an external camera image captured at the current time t(k) are shown. Furthermore, it is assumed that at the past time t(k-1), among the outlines C0 to C4, outlines C0 and C2 overlap, and further outlines C1 and C2 overlap to form the track boundary contour line, and outlines C3 and C4 overlap to form the track boundary contour line. Furthermore, it is assumed that at the current time t(k), among the outlines C5 to C9, outlines C6 and C7 overlap, outlines C7 and C8 overlap, and further outlines C8 and C6 overlap to form the track boundary contour line.

[0065] Here, in the contour integration based on the time axis described above, if the extraction unit 141 determines that contours C2 and C5, contours C4 and C6, and contours C4 and C8 are the same contour, it integrates contours C0, C1, C2, and C5, and further integrates contours C3, C4, C6, C7, and C8. This makes it possible to recognize the contour shape of the track boundary based on the passage of time more appropriately. Furthermore, by storing the results of contour integration based on the spatial axis and contour integration based on the time axis, or the contour overlap information in each integration process, in the storage device 70, the extraction unit 141 can utilize some of the overlap information in the next integration process, thereby speeding up the processing.

[0066] By performing the above-described process, and by simplifying the contour lines through contour line integration processing as shown in Figure 9 or Figure 10, the extraction unit 141 can reduce the subsequent processing load and speed up processing.

[0067] [Lane Recognition Unit] Next, the functions of the lane recognition unit 142 will be described in detail. Figure 11 is a diagram illustrating the functions of the lane recognition unit 142. The lane recognition unit 142 performs lane instance processing to recognize the position and shape of a partial lane (trail) based on the information of the integrated result of contour lines extracted by the extraction unit 141 (for example, the position of the contour lines of the track boundary).

[0068] To explain in more detail, first the lane recognition unit 142 receives information on the result of integrating multiple contour lines extracted by the extraction unit 141, and then searches for the center points of two of the input contour lines.

[0069] [Center Point Search] Figure 12 is a diagram illustrating center point search. In the example in Figure 12, a track boundary TB1 consisting of contour points P0 to P3 and a track boundary TB4 consisting of contour points P4 to P17 are shown. Also in the example in Figure 12, contour line segments LS0 to LS3 are shown, which are formed by connecting contour points P0 and P1, P1 and P2, P2 and P3, and P3 and P0 of track boundary TB1 with straight lines in a clockwise direction. Similarly, in track boundary TB4, contour line segments LS4 to LS17 are shown, which are formed by connecting each point between contour points P4 to P17 with straight lines in a clockwise direction.

[0070] The lane recognition unit 142 extracts, for example, the pair of contour segments that are closest to each other, and are facing each other and substantially parallel. "Facing each other" means that the two contour segments are opposite each other, for example, when the endpoint of one of the two contour segments is projected onto the other contour segment, at least one of the projection points lies on the contour segment onto which it is projected. "Substantially parallel contour segments" means, for example, contour segments whose angle formed by the extension directions of the two contour segments is less than a predetermined angle. In addition to the above conditions, the lane recognition unit 142 may also include (or add to) the following: the two contour segments are substantially parallel and the contour segments are of a predetermined length or longer, the contour segments are located to the left and right of the position of the moving body 1, and the contour segments are not extracted from the contour of the same lane boundary.

[0071] Figure 13 is a diagram illustrating a method for searching for pairs of contour segments. As an example, Figure 13 shows contour segment LS1, which connects contour points P1 and P2 shown in Figure 12 with a straight line, and contour segment LS17, which connects contour points P4 and P17 with a straight line. When the lane recognition unit 142 searches for pairs of contour segments based on contour segment LS1 of the track boundary TB1, for example, LS5 of the track boundary TB4 which is different from contour segment LS1, LS1 and LS6, LS1 and LS7, ..., LS1 and LS17, the following processing (1) to (4) is performed for each pair of contour segments.

[0072] (1) The lane recognition unit 142 sets point Q0 by projecting (moving laterally) contour point P17 onto contour line segment LS1, point Q1 by projecting contour point P4 onto contour line segment LS1, point Q2 by projecting contour point P2 onto contour line segment LS17, and point Q3 by projecting contour point P1 onto contour line segment LS17. However, contour line segment LS17 is further extended in the extending direction of contour line segment LS1 so that it can be projected from contour points P1 and P2. (2) The lane recognition unit 142 derives the width corresponding to each set projection point. Specifically, the lane recognition unit 142 derives the distance LW0 from contour point P17 to point Q0, the distance LW1 from contour point P4 to point Q1, the distance LW2 from contour point P2 to point Q2, and the distance LW3 from contour point P1 to point Q3. Here, in the example in Figure 13, both points Q0 and Q1 lie on the actual contour line segment LS1 of the projection target. Therefore, the lane recognition unit 142 determines that contour line segments LS1 and LS17 are opposite each other because at least one of the projection points Q0 to Q3 lies on the line segment of the projection target. (3) Since the angle formed by contour line segments LS1 and LS17 is less than a predetermined angle, the unit determines that contour line segments LS1 and LS17 are approximately parallel. (4) The lane recognition unit 142 sets the narrower of the widths LW0 and LW1 corresponding to points Q0 and Q1 as the width corresponding to the pair of contour line segments (LS1, LS17).

[0073] In addition, in (4) above, "the narrower one" may be replaced with "the wider one," or the "average" of width LW0 and width LW1 may be used. After performing the above processing, the lane recognition unit 142 compares the widths extracted in (4) above for pairs of contour segments that satisfy the conditions in (2) and (3) above, extracts the shortest (smallest width) or pair of contour segments (LS1, LS17) whose width is less than the threshold, and searches for the center point of the lane (road) between the pair of contour segments, treating the two extracted contour segments as a pair.

[0074] Furthermore, if the lane recognition unit 142 recognizes at least three contour line segments extending in the same direction, it may perform the above-described process on adjacent contour lines among the at least three contour lines and search for the center point of a pair of contour line segments that satisfy the above-described conditions.

[0075] Figure 14 is a diagram illustrating the method for finding the center point. In the example in Figure 14, the orientations of contour line segments LS1 and LS17 are aligned, and the resulting contour line segments are designated as LS1' and LS17'. Next, the lane recognition unit 142 finds the center points of the starting points and ending points of the aligned contour line segments. Specifically, as shown in Figure 14, the lane recognition unit 142 derives the center point MP0 of the starting point (contour point P1) of contour line segment LS1' and the starting point (contour point P4) of contour line segment LS17', and the center point MP1 of the ending point (contour point P2) of contour line segment LS1' and the ending point (contour point P17) of contour line segment LS17'. Then, the lane recognition unit 142 generates points at predetermined intervals (for example, equal intervals) on the line segment CLS connecting the center points MP0 and MP1. Furthermore, if you want to project the generated points onto contour segment LS1' and contour segment LS17', the points that exist on those lines are designated as center points CP. In the example in Figure 14, center points CP0 to CP4 are generated as a result of the search.

[0076] Furthermore, the lane recognition unit 142 recognizes attribute information for the searched center points CP0 to CP4, including the direction of the lane at that point, i.e., the direction of the line segment CLS, and information about the contour line segment to the left (LS17') and the contour line segment to the right (LS1') with respect to the direction of the line segment CLS. The attribute information for center points CP0 to CP4 includes, for example, position information (X, Y, Z), the extension direction of the lane demarcated by the two contour line segments (direction of the line segment CLS), the contour line segment to the left and the contour line segment to the right with respect to the extension direction, and identification information (lane ID) that identifies the lane (road). The attribute information of the center points described above may be stored in the storage device 70.

[0077] Returning to Figure 11, the lane recognition unit 142 then performs clustering of the searched (generated) center points. Figure 15 is a diagram illustrating the clustering of center points. In the example in Figure 15, the center points CP searched by the center point search described above are shown for contour line segments between contour points of the track boundaries TB1 to TB5.

[0078] The lane recognition unit 142, for example, clusters (groups) the searched center points CP (center point group) based on the positional information of each center point. For example, the lane recognition unit 142 clusters point groups where the distance between two center points is within a predetermined distance based on Euclidean distance, and assigns identification information (for example, a sequential lane ID) to each cluster. In the clustering process, by setting the predetermined distance (the maximum distance between two points to be in the same cluster) to an appropriate value, for example, center point groups before and after intersections or center point groups before and after junctions can be classified as different clusters. An appropriate value may be a value corresponding to a predetermined road shape such as an intersection or junction, or it may be a fixed value. Furthermore, when the lane recognition unit 142 performs clustering of center point groups, it may also add conditions such as the direction in which multiple center points are aligned or the distance between contour lines where the center points were searched (lane direction and width) when performing clustering.

[0079] In the example shown in Figure 15, cluster G0 of the center point cloud behind the moving object 1 and clusters G1 to G4 of the center point cloud in front of it are extracted. These clusters may be recognized as center point clouds of different lane units (partial lanes) based, for example, on their position relative to the moving object 1 or their direction of extension.

[0080] Next, the lane recognition unit 142 aligns the orientation of the center points CP along the lanes for each cluster G0 to G4. Figure 16 shows the state before and after the process of aligning the orientation of the center points. In the example in Figure 16, the state before (Figure 16-A) and after (Figure 16-B) the process of aligning the orientation of the group of center points when there are seven center points CP0 to CP6 in one cluster is shown. The orientation of the center points CP is the direction of the lane assigned to each point when the center points were generated.

[0081] For example, the lane recognition unit 142 compares the nearest center points CP1 and CP2 with the center point CP0 as a reference. Since center point CP1 has the same orientation as center point CP0, the orientation of CP2 is aligned with this orientation. The lane recognition unit 142 then aligns the next center points CP6 and CP2 with the same orientation as center point CP0, and further aligns center points CP4 and CP3 with the same orientation as center point CP0, etc. Note that the position of the initial reference point is not limited to center point CP0, but may be any other point. This allows the pair of contour line segments corresponding to the center point (the contour line segments on the left and right sides of the lane) to be clustered (grouped) in a process described later. Note that when the lane recognition unit 142 aligns the orientation of the center points CP, it may align them based on, for example, the direction of the X-axis or Y-axis of the moving object coordinate system. Alternatively, the lane recognition unit 142 may, instead of the method described above, sequentially align the orientations, for example, by aligning the orientation of center point CP1 with center point CP0, then aligning the orientation of center point CP6 with center point CP1, then aligning the orientation of CP5 with center point CP6, then aligning the orientation of center point CP4 with center point CP5, and so on. By sequentially aligning the orientations in this way, the orientation of the center points can be aligned even on a U-shaped road, for example.

[0082] Returning to Figure 11, the lane recognition unit 142 then tracks the center points CP. Specifically, the lane recognition unit 142 clusters the center points generated in the past and the center points generated in the present. This allows the lane ID assigned in the past to be inherited from the center points in the past to the center points in the present. Furthermore, when the lane recognition unit 142 aligns the directions of the center points CP in the past and present, it uses the orientation of the oldest center point as a reference, thereby inheriting the lane direction determined in the past to the present.

[0083] Returning to Figure 11, the lane recognition unit 142 then performs clustering of contour line segments in accordance with the clustering results of the tracked center point cloud. Figure 17 is a diagram illustrating the clustering of contour line segments. In the example in Figure 17, each clustered cluster G0 to G4 of the center points (i.e., each lane) is shown in a different display manner. Different display manners include, for example, different colors, patterns, line types, or differences indicated by characters, etc.

[0084] The lane recognition unit 142 assigns the same lane ID as the center point to the contour line segment that was the source of each center point for each cluster G0 to G4. In the example in Figure 17, the contour line segment LSG0 that was the source of each center point for cluster G0, the contour line segment LSG1 that was the source of each center point for cluster G1, the contour line segment LSG2 that was the source of each center point for cluster G2, the contour line segment LSG3 that was the source of each center point for cluster G3, and the contour line segment LSG4 that was the source of each center point for cluster G4 are each assigned the same lane ID as the center point.

[0085] Next, the lane recognition unit 142 aligns the orientation of the center point for each lane ID and groups the left and right contour line segments of the lane. Note that, as a result of this process, it is possible that two or more lane IDs may be assigned to a single contour line segment, as shown in Figure 17, for example.

[0086] Returning to Figure 11, the lane recognition unit 142 then connects the left and right contour line segments in each lane. For example, if the distance between contour line segments is less than a predetermined distance, the lane recognition unit 142 connects the contour line segments with a straight line or a line corresponding to the shape of the contour line segments. Alternatively (or in addition to the above condition), the lane recognition unit 142 may connect two contour line segments if the angle formed by adjacent line segments in different lanes is less than a predetermined angle. This allows the position and shape of each lane (lane instance) to be recognized.

[0087] Furthermore, the lane recognition unit 142 may extract only the sections from the connection results where the projected point, when the center point is projected onto the track boundary corresponding to the contour line segment, overlaps with the track boundary line, and construct the left and right track boundaries of each lane. In addition, the lane recognition unit 142 can be made robust against undetected parts of the track boundary, noise, or occlusion by storing information on contour line segments observed in the past in the storage device 70. Furthermore, when the lane recognition unit 142 stores contour line segments in the storage device 70 and reuses them during repeated processing, the connection relationships of the contour line segments can also be stored in the storage device 70, thereby speeding up the processing up to the connection of the contour line segments.

[0088] [Connection Section] Next, the function of the connection section 143 will be explained. Figure 18 is a diagram illustrating the details of the function of the connection section 143. The connection section 143 recognizes the connection relationship of each lane based on the connection results of the contour line segments recognized by the lane recognition section 142 (the position and shape of each lane). For example, the connection section 143 enumerates connection candidates for each lane and further optimizes the connection relationship for the enumerated connection candidates.

[0089] [Enumeration of Connection Candidates] Figure 19 is a diagram illustrating the enumeration of connection candidates. In the example in Figure 19, the center points CP0 to CP6 included in a single lane (lane instance) LI1 and the left and right contour line segments LS-L and LS-R corresponding to the center points CP0 to CP6 are shown. The connection section 143 rearranges the center points belonging to the lane on the path axis, and the first and last center points become the start and end points of the lane.

[0090] Specifically, as shown in Figure 19, the connection unit 143 selects an arbitrary center point (for example, center point CP0) from among the center points CP0 to CP6 belonging to lane LI1. Next, the connection unit 143 defines a hyperplane HP that includes center point CP0 and is perpendicular to the direction of center point CP0, and rearranges the center points CP1 to CP6 based on the distance to the hyperplane HP. Then, the connection unit 143 sets the first center point CP3 as the starting point of lane LI1 and the last center point CP4 as the ending point of lane LI1. Alternatively, the connection unit 143 may set center points corresponding to the starting and ending points based on the relative positions and directions of the respective center points CP0 to CP6.

[0091] Next, the connection unit 143 determines connecting line segments that connect to the start and end points of other lanes based on the information of the start and end points of lane LI1. Figure 20 is a diagram illustrating how to determine the connecting line segments. In the example in Figure 20, lane LI1 and lane LI2, to which the center points CP7 to CP11 belong, are shown. The start and end points of lane LI2 are the center points CP11 and CP7. For example, the connection unit 143 determines the shortest connecting line segment for each combination of connecting line segments that connect the start and end points of each lane. The connection unit 143 also lists connecting line segments that do not overlap with lanes as candidate connecting line segments.

[0092] For example, as shown in Figure 20, when finding a connecting line segment between lanes LI1 and LI2, first, candidate connecting line segments are extracted based on the combination of endpoints (start point, end point) of lanes LI1 and LI2. In the example in Figure 20, candidate connecting line segments LC0 connecting center points CP11 and CP3, candidate connecting line segments LC1 connecting center points CP11 and CP4, candidate connecting line segments LC2 connecting center points CP7 and CP3, and candidate connecting line segments LC3 connecting center points CP7 and CP4 are extracted.

[0093] Next, the connection unit 143 selects the shortest of the four candidate connection segments LC0 to LC3 described above. In the example in Figure 20, candidate connection segment LC2 is selected as the shortest. The connection unit 143 performs the above process for several other lanes to confirm that candidate connection segment LC2 does not overlap with any other lane. If there is no overlap, the connection unit 143 lists it as a candidate connection segment.

[0094] Figure 21 is a diagram illustrating the overlap between candidate connection segments and other lanes. In the example in Figure 21, there are three different lanes (lane instances) LI1, LI2, and LI3, and in addition to the candidate connection segment LC2 mentioned above, the shortest candidate connection segment LC4 between lanes LI2 and LI3 is shown.

[0095] When a candidate connecting line segment overlaps with another lane, it means, for example, as shown in Figure 21, that the region of the polygon (the contour shape for each lane) formed by the contour line segments and endpoints of each lane overlaps with the connecting line segment. In the example in Figure 21, candidate connecting line segment LC4 overlaps with the polygon region of another lane and is therefore not listed as a candidate connecting line segment. On the other hand, candidate connecting line segment LC2 does not overlap with the polygon region of another lane and is therefore listed as a candidate connecting line segment.

[0096] Furthermore, when the connection unit 143 enumerates candidate connection lines, it may add conditions such as the length and direction of the candidate connection lines in addition to (or instead of) the conditions described above. In this case, the connection unit 143 enumerates candidate connection lines, for example, when the length of the candidate connection line is less than a predetermined distance, and / or when the extension direction of the candidate connection line is within a predetermined angular range based on the arrangement direction of the center point group belonging to the connected lanes. This makes it possible to suppress the connection of lanes that are far apart or lanes that are difficult to imagine as part of the track shape. In addition, the connection unit 143 may, for example, determine the overlap between a candidate connection line and other lanes, or if a drivable area has been detected in advance, determine the overlap between that area and the candidate connection line. Furthermore, when the connection unit 143 enumerates candidate connection lines, it can store the information of the connection lines in the storage device 70 and reuse it when enumerating candidates next time, thereby improving the efficiency of the process.

[0097] Returning to Figure 18, the connection unit 143 then optimizes the lane connection relationships based on the information of the candidate connection segments. Figure 22 is a diagram illustrating the optimization of connection relationships. In the example in Figure 22, three different lanes LI1 to L13 and candidate connection segments LC2, LC5, and LC6 between the lanes are shown. Candidate connection segment LC5 is a candidate connection segment between lanes LI1 and LI3, and candidate connection segment LC6 is a candidate connection segment between lanes LI2 and LI3.

[0098] For example, the connection unit 143 represents each lane LI1 to LI3 as a node and the connection relationship as an edge. The connection unit 143 then creates a graph in which the lengths of candidate connection segments are used as edge weights. For example, the connection unit 143 solves the shortest path problem between all lanes using the Warshall-Floyd algorithm and then excludes candidate connection segments that are not used in any shortest path. In the example in Figure 22, the sum of the lengths of candidate connection segment LC2 and candidate connection segment LC5 is compared with the length of candidate connection segment LC6, and candidate connection segment LC6 is deleted if the sum is smaller than the length of candidate connection segment LC6. The connection unit 143 may also perform an optimization process to merge two lanes connected by candidate connection segments whose length is less than a threshold. This determines the final connection segment.

[0099] [Road Shape Recognition Unit] The road shape recognition unit 144 recognizes the road shape around the moving body 1 based on the information indicating the connection relationship by the connecting line segments described above. In this case, the road shape recognition unit 144 may also recognize the positional relationship of the left and right contour line segments of the lanes connected by the connecting line segments, and may connect the contour line segments with curves or straight lines based on the shape of the contour line segments and the arrangement of the center points, thereby recognizing the contour line of the road (the boundary line that demarcates the road). Furthermore, the road shape recognition unit 144 recognizes the road shape more specifically based on, for example, the connection state of the lanes, the position of the center points, the positional relationship of the contour lines belonging to the connected lanes, etc.

[0100] For example, the road shape recognition unit 144 distinguishes and recognizes a first-direction contour line extending in a first direction from the contour line and a second-direction contour line extending in a second direction different from the first direction, and recognizes the road shape around the moving body 1 based on the multiple contour lines that have been distinguished and recognized. The road shape recognition unit 144 also recognizes whether there is an intersection or a branch road in the road shape based on the angle formed by the first direction and the second direction. For example, if the angle is less than a predetermined angle, it recognizes that there is a branch road, and if it is within a predetermined angle range (for example, about 70 to 110 degrees), it recognizes that there is an intersection. The road shape recognition unit 144 may also recognize intersections based on the positional relationship between the positional information of each lane and the part where the center point of the lane does not exist.

[0101] Furthermore, in the embodiment, the recognition unit 140 may, for example, when it recognizes a pair of first directional contour lines or a pair of second directional contour lines, search for the center point of the two recognized contour lines and recognize the road shape around the moving body 1 based on the searched center point. This makes it possible to perform the above-described processing in situations where there is a high probability of intersections etc., and not perform the above-described processing in other recognition results, thereby reducing the overall processing load.

[0102] Furthermore, in the embodiment, the recognition unit 140 may, for example, when it recognizes a pair of first directional contour lines or a pair of second directional contour lines, search for the center point of the two recognized contour lines and recognize them as the center point of the two contour lines if the number of center points at a predetermined distance is greater than or equal to a predetermined number. This makes it possible to recognize lanes with a length of a predetermined distance or more.

[0103] Furthermore, the road shape recognition unit 144 may, for example, recognize a road demarcated by a pair of contour lines (left and right contour lines) corresponding to a lane as a Type 1 road if the distance between the center of the contour line and the contour line is greater than or equal to a threshold, and recognize a road defined by the pair of contour lines as a Type 2 road, which is different from a Type 1 road, if the distance is less than the threshold. A Type 1 road is, for example, a carriageway, and a Type 2 road is, for example, a sidewalk, but each may be a different type depending on the road width. This makes it possible to recognize the road type more accurately based on the distance between the pair of contour lines.

[0104] Furthermore, if the road shape recognition unit 144 recognizes multiple Class 1 roads, the HMI control unit 164 may cause the display unit of the HMI 30 to display image information representing each of the multiple Class 1 roads in a different display manner.

[0105] Figure 23 shows an example of an image displayed in the embodiment. The image IM30 shown in Figure 23 shows lanes LI11 to LI14 located in front of the moving body 1 (in the direction of travel) and the center point CP belonging to each lane. In the example in Figure 23, when the road shape recognition unit 144 recognizes the contour line (contour line portion) of lane LI11 as the first direction contour line, it recognizes the boundary lines of lanes LI12, LI13, and LI14 as the second direction contour line. Furthermore, based on the distance between the center point CP and the contour line, it recognizes lanes LI11 to LI13 as roadways (an example of a first-class road) and lane LI14 as a sidewalk (an example of a second-class road). In addition, the road shape recognition unit 144 recognizes that there is an intersecting road in front of the moving body 1 because the angle formed by the first direction contour line and the second direction contour line is within a predetermined angle range. Furthermore, the road shape recognition unit 144 may recognize an area (region) without a center point where lanes LI11, LI12, and LI13 are connected as an intersection.

[0106] When the HMI control unit 1 notifies the occupant D of the recognized mobile body 1 of the road shape present around the mobile body 1, it generates images corresponding to lanes LI11 to LI14 (for example, images showing outlines (lane lines)) in addition to an image showing the mobile body 1 and an image showing the direction of travel, as shown in Figure 23. In the example in Figure 23, since multiple Type 1 roads are recognized, the HMI control unit 164 generates images displaying each of the Type 1 roads in a different display manner, such as different colors, patterns, or line types. The HMI control unit 164 may also generate an image displaying lane LI14, which is a Type 2 road (sidewalk), in a different display manner than the Type 1 roads.

[0107] Figure 24 shows another example of an image displayed in the embodiment. The image IM40 shown in Figure 24 shows lanes LI21 to LI25 located in front of the moving body 1 (in the direction of travel) and the center point CP belonging to each lane. In the example in Figure 24, when the road shape recognition unit 144 recognizes the contour lines (contour segments) of lanes LI21 and LI22 as first direction contour lines, it recognizes the contour lines of lanes LI23 and L24 as second direction contour lines. Furthermore, based on the distance between the center point CP and the contour lines, it recognizes lanes LI21 to LI24 as roadways (an example of a first-class road) and lane LI25 as a sidewalk (an example of a second-class road). In addition, the road shape recognition unit 144 recognizes that there is a branch road in front of the moving body 1 because the angle formed by the first direction contour line and the second direction contour line is less than a predetermined angle. Furthermore, the road shape recognition unit 144 may recognize a portion (region) where there is no center point CP in the area where lanes LI21, LI22, and LI23 are connected as a branching point.

[0108] When the HMI control unit 1 notifies the occupant D of the recognized mobile body 1 of the road shape present around the mobile body 1, it generates images corresponding to lanes LI21 to LI25 (for example, images showing outlines (lane lines)) in addition to an image showing the mobile body 1 and an image showing the direction of travel, as shown in Figure 24. In the example of Figure 24, since multiple Class 1 roads are recognized, the HMI control unit 164 generates images displaying each of the Class 1 roads in a different display manner, such as different colors, patterns, or line types. The HMI control unit 164 may also generate an image displaying lane LI25, which is a Class 2 road (sidewalk), in a different display manner than the Class 1 roads.

[0109] The HMI control unit 164 generates images as shown in Figures 23 and 24 and displays them on the HMI 30, etc., thereby informing the occupant D that the recognition unit 140 has distinguished and recognized multiple Class 1 roads, and also informing the occupant D that Class 1 roads and Class 2 roads have been distinguished and recognized. Furthermore, it is possible to provide the occupant D with more appropriate information regarding the recognition status by the recognition unit 140.

[0110] For example, as shown in Figures 23 and 24, if there are multiple Type 1 roads in the direction of travel of the mobile body 1, the inquiry unit 166 may inquire with the occupant of the mobile body 1 about which of the multiple Type 1 roads the mobile body 1 should take. In this case, in addition to displaying images IM30 and IM40 as shown in Figures 23 and 24 on the HMI 30, the inquiry unit 166 may also generate text images or audio information prompting the occupant to select one of the multiple Type 1 roads in the image, and obtain the generated information from the HMI 30. Note that the above inquiry is executed, for example, when the mobile body 1 is performing a predetermined driving control (for example, automatic driving such as LKAS).

[0111] By making the above-described inquiry, the occupant D of the mobile body 1 can indicate the direction they wish to go (the path of the mobile body 1) using the HMI 30 or the control device 14 (for example, by operating the steering wheel WH).

[0112] When the control unit 162 receives instructions from the occupant regarding the direction of travel of the mobile body 1, it controls the movement of the mobile body 1 (for example, straight-line control, right / left turn control, turning control, etc.) so that it moves (travels) along the Type 1 road selected by the occupant's instructions. This allows the mobile body 1 to travel in accordance with the occupant's requirements and also allows the control to continue even if there are multiple Type 1 roads ahead.

[0113] In this embodiment, the road shape recognition unit 144 may recognize that a discontinuous center point is an area requiring deceleration control (including stopping control) of the moving body 1 when a second directional contour line extending in a different direction from the first directional contour line is recognized at the discontinuation point. In the example in Figure 23, the intersection is an area requiring deceleration control, and in the example in Figure 24, the branching point is an area requiring deceleration control. The driving control unit 162 performs deceleration control based on these recognition results. Therefore, more appropriate driving control can be performed based on the recognized road shape.

[0114] Furthermore, if no instruction input is received from the occupant within a predetermined time after the inquiry is made, the inquiry unit 166 may instruct the driving control unit 162 to proceed to a predetermined road among several Class 1 roads (for example, the road with the widest width or the road after turning left), or it may perform control to move to a safe position (for example, the edge of the road) and stop.

[0115] The object recognition unit 145 recognizes objects present around the moving body 1 (for example, other vehicles, pedestrians and other traffic participants, and other obstacles). The object recognition unit 145 also recognizes the position (relative position) and speed (relative speed) of each object. The driving control unit 162 may drive the vehicle along the road shape, or it may perform driving control that controls steering or speed to avoid contact with objects.

[0116] In addition to the processing described above, the recognition unit 140 may also analyze the camera image from the internal camera 16 to recognize the state of the occupant D of the mobile vehicle 1. In this case, the recognition unit 140 performs known image analysis processing on the camera image from the internal camera 16 and, based on the analysis results, recognizes whether the occupant D is in a state unsuitable for driving the mobile vehicle 1, such as dozing off or looking away, based on the orientation of the occupant D's face, the state of their eyes, the posture of their body, etc., included in the image. This recognition result is output to the driving control unit 162, and the driving control unit 162 executes driving control according to the state of the occupant D.

[0117] [Processing Flow] Next, the processing flow executed by the mobile body control device of the embodiment will be described. Figure 25 is a flowchart showing an example of the processing flow executed by the control device 100 of the embodiment. In the example of Figure 25, among the various processes executed by the control device 100, the processing that recognizes the road shape around the mobile body based on the contour points and contour line segments of the road boundary and controls the movement of the mobile body based on the recognition result will be described in particular. Note that the processing in Figure 25 may be executed repeatedly at predetermined timings.

[0118] In the example shown in Figure 25, the acquisition unit 120 acquires an image captured by the external camera 11 (step S100). In step S100, the acquisition unit 120 may generate a mask image from the acquired image. The extraction unit 141 extracts contour points from the mask image (step S110). Next, the extraction unit 141 converts the coordinate system of the mask image (image coordinate system) to the moving object coordinate system (homography transformation) (step S120) and performs contour simplification processing (step S130). Next, the extraction unit 141 performs processing to integrate the simplified contour lines (step S140).

[0119] Next, the lane recognition unit 142 searches for the center point of a pair of contour lines among the contour lines (integrated contour lines) extracted by the extraction unit 141 (step S150), and performs clustering of the center points based on the positions of the searched center point group (step S160). Next, the lane recognition unit 142 tracks the center points based on the past center point search results (clustering results) and the current center point search results (step S170), and performs clustering of contour lines based on the tracking results to recognize partial lanes (step S180).

[0120] Next, the connection unit 143 enumerates the connection line segments between partial lanes (step S190) and optimizes the connection relationships for the enumerated connection line segments (step S200). Next, the road shape recognition unit 144 recognizes the road shape based on the optimized lane connection relationships (step S210). In this road shape recognition, for example, it distinguishes and recognizes first direction contour lines extending in a first direction from multiple contour lines and second direction contour lines extending in a second direction different from the first direction, and recognizes the road shape around the moving body based on the multiple recognized contour lines. Next, the object recognition unit 145 recognizes objects around the moving body 1 (step S220). Next, the driving control unit 162 executes driving control (for example, automatic driving such as LKAS) according to the recognized road shape and surrounding objects (step S230). This completes the processing of this flowchart.

[0121] [Modification] In this embodiment, instead of (or in addition to) using an external camera image captured by an external camera 11 (an example of an imaging unit) mounted on the mobile body 1 to search for the center point of a pair of contour line segments and perform road shape recognition, for example, images captured by a fixed camera installed near the road around the mobile body 1 may be acquired via a communication device 20, or images captured by another mobile body traveling in front of or behind the mobile body 1 may be acquired, and similar processing may be performed using the acquired images.

[0122] Furthermore, according to the embodiment, the HMI control unit 164 may generate (or output from the HMI 30) image IM10 shown in Figure 5, image IM20 shown in Figure 7, and other information related to the recognition of the surrounding conditions of the mobile body 1, in addition to (or instead of) the images IM30 and IM40 shown in Figures 23 and 24. In addition, the inquiry unit 166 may make inquiries about the route described above, as well as inquiries about the driving control that the mobile body 1 will execute based on the recognition results by the recognition unit 140, and inquiries for switching the driving control that is currently being executed.

[0123] In this embodiment, the object recognition unit 145 may also recognize landmarks on the road surface (for example, pedestrian crossings). In this case, the connection unit 143 may recognize the location and shape of the road or distinguish between Type 1 roads (for example, carriageways) and Type 2 roads (for example, sidewalks) based on the location and shape of the pedestrian crossings recognized by the object recognition unit 145. By using information on landmarks in this way, the road shape can be recognized with greater accuracy.

[0124] In addition, in the embodiment, the recognition unit 140 may recognize the road shape around the mobile body 1 based on the road shape recognized by the method described above and the road shape obtained from the map information 72 stored in the storage device 70. In this case, the recognition unit 140 recognizes the road shape around the mobile body 1 by referring to the map information 72 based on the position information of the mobile body 1 measured by the positioning device 18, and recognizes the final road shape based on the matching result between the recognized road shape and the road shape recognized from the external camera image. Furthermore, the recognition unit 140 may correct at least a part of the road shape obtained from the external camera image based on the road shape obtained from the map information 72. This makes it possible to recognize the road shape with higher accuracy.

[0125] In this embodiment, the mobile vehicle 1 may be an unmanned vehicle that can be driven by a remote device such as a user's. In this case, the information (images and sounds) generated by the HMI control unit 164 may be transmitted to the user's terminal device (e.g., a smartphone or tablet) via the communication device 20 instead of being output to the HMI 30 (or in addition to being output to the HMI 30). This allows the user to more accurately grasp the road shape around the mobile vehicle 1 from a location away from the mobile vehicle 1.

[0126] According to the embodiments described above, the mobile body control device includes an acquisition unit 120 that acquires an image including the road surface around the mobile body 1, an extraction unit 141 that extracts the contour lines of the road boundary of the mobile body 1 from the image, and a road shape recognition unit that distinguishes and recognizes a first direction contour line extending in a first direction from the contour lines extracted by the extraction unit 141 and a second direction contour line extending in a second direction different from the first direction, and recognizes the road shape around the mobile body 1 based on the recognized plurality of contour lines. This allows for more favorable recognition of the road of the mobile body 1 and more appropriate control of the mobile body.

[0127] Specifically, according to the embodiment, for example, by using information on the center points of paired contour segments, the road shape can be recognized with higher accuracy. Furthermore, according to the embodiment, even without using map information, the road environment can be recognized using semantic segmentation results output from a DNN or the like, and road shapes such as intersections and junctions can be recognized, so that more appropriate driving control can be performed based on the recognized road shape information.

[0128] For example, according to the embodiment, the position and connection relationships of partial lanes can be recognized from road boundary information detected by a DNN or the like, enabling autonomous driving (automatic operation) of the mobile body 1 in places with multiple lanes, such as intersections and roadways. Furthermore, according to the embodiment, by focusing on the center point of the lane corresponding to the contour line segment and utilizing the characteristic that the center point of the lane is interrupted at intersections, lanes can be clustered. This allows for accurate recognition of the number and orientation of lanes, and by geometrically optimizing the connection relationships of each lane, the connection relationships can be understood more accurately from road boundary information obtained from external camera images alone.

[0129] In particular, since the mobile body 1 used in this embodiment is a mobility device that can travel not only on roadways but also on sidewalks, etc., the above processing allows for more accurate recognition of road shapes, enabling the mobile body 1 to respond more effectively to all driving conditions.

[0130] The embodiment described above can be expressed as follows: A mobile body control device comprising: a storage medium for storing computer-readable instructions; a processor connected to the storage medium, wherein the processor executing the computer-readable instructions to: acquire an image including the road surface around a mobile body; extract contour lines of the road boundary of the mobile body from the image; distinguish and recognize first direction contour lines extending in a first direction from the extracted contour lines and second direction contour lines extending in a second direction different from the first direction; and recognize the road shape around the mobile body based on the recognized plurality of contour lines.

[0131] Although embodiments for carrying out the present invention have been described above using examples, the present invention is not limited in any way to these embodiments, and various modifications and substitutions can be made without departing from the spirit of the present invention.

[0132] 10... External detection device, 11... External camera, 12... Mobile sensor, 14... Operator, 16... Internal camera, 18... Positioning device, 20... Communication device, 30... HMI, 40... Drive unit, 50... Movement mechanism, 70... Memory device, 100... Control device, 120... Acquisition unit, 140... Recognition unit, 141... Extraction unit, 142... Lane recognition unit, 143... Connection unit, 144... Road shape recognition unit, 145... Object recognition unit, 160... Control unit, 162... Driving control unit, 164... HMI control unit, 166... ​​Inquiry unit

Claims

1. A mobile body control device comprising: an acquisition unit that acquires an image including the road surface around a mobile body; and a recognition unit that extracts the contour lines of the road boundary of the mobile body from the image, distinguishes and recognizes a first direction contour line extending in a first direction from the extracted contour line and a second direction contour line extending in a second direction different from the first direction, and recognizes the road shape around the mobile body based on the recognized plurality of contour lines.

2. The mobile body control device according to claim 1, wherein the recognition unit, upon recognizing a pair of first directional contour lines or a pair of second directional contour lines, searches for the center point of the two recognized contour lines and recognizes the road shape based on the center point that was searched.

3. The mobile body control device according to claim 2, wherein, in the search for the center point, if the continuous search for the center point is interrupted and a second directional contour line extending in a direction different from the first directional contour line is recognized at the interrupted location, the recognition unit recognizes that the interrupted location is an area requiring deceleration control of the mobile body.

4. The mobile body control device according to claim 2, wherein the recognition unit, upon recognizing a pair of first directional contour lines or a pair of second directional contour lines, searches for the center point of the two recognized contour lines, and recognizes that the number of center points at a predetermined distance is equal to or greater than a predetermined number.

5. The mobile body control device according to claim 1, wherein the recognition unit recognizes that a road formed by a pair of contour lines is a Type 1 road when the distance between the center of a pair of contour lines and the contour line is greater than or equal to a threshold, and recognizes that a road formed by a pair of contour lines is a Type 2 road when the distance is less than the threshold.

6. The mobile device control device according to claim 5, further comprising a display control unit that, when the recognition unit recognizes a plurality of the first type roads, outputs each of the plurality of the first type roads to the display unit in a different display manner.

7. The mobile body control device according to claim 5, further comprising an inquiry unit that, when a plurality of the first type roads are recognized by the recognition unit, inquires with the occupant of the mobile body which first type road from among the plurality of first type roads will be used as the route for the mobile body.

8. The mobile body control device according to claim 5, further comprising a driving control unit that, when a plurality of the first type roads are recognized by the recognition unit, moves the mobile body to the first type road selected by an instruction from the occupant operating the mobile body.

9. The mobile body control device according to any one of claims 1 to 8, wherein the recognition unit, upon recognizing at least three or more contour lines extending in the same direction, searches for the center points of adjacent contour lines among the at least three or more contour lines, determines a pair of contour lines based on the positions of the searched center points, and recognizes the road shape around the mobile body based on the positions of the determined pair of contour lines.

10. The mobile body control device according to any one of claims 1 to 8, wherein the recognition unit integrates the contour lines of the road boundary extracted from each of the plurality of images based on position information when the acquisition unit has acquired a plurality of images of the mobile body taken in different directions.

11. The mobile body control device according to any one of claims 1 to 8, wherein the recognition unit, when multiple images captured by the acquisition unit at different times are acquired, compares contour lines acquired from each image, and if the distance between contour lines is less than a threshold and the extension direction of each contour line is within a predetermined range, determines that the compared contour line segments are the same contour and integrates them.

12. A mobile body control device comprising: an acquisition unit that acquires an image including the road surface around a mobile body; a lane recognition unit that extracts contour lines of the road boundary of the mobile body from the image, searches for the center points of pairs of contour line segments among the extracted contour lines, performs clustering based on the positions of the searched center point cloud, and recognizes the position and shape of a partial lane based on the position information of the clustered center point cloud; and a recognition unit that recognizes the road shape based on the position and shape of the lane recognized by the lane recognition unit.

13. A method for controlling a moving object, comprising: a computer acquiring an image including the road surface around the moving object; extracting contour lines of the road boundary of the moving object from the image; distinguishing and recognizing first direction contour lines extending in a first direction from the extracted contour lines and second direction contour lines extending in a second direction different from the first direction; and recognizing the road shape around the moving object based on the recognized multiple contour lines.

14. A program that causes a computer to acquire an image including the road surface around a moving object, extract contour lines of the road boundary of the moving object from the image, distinguish and recognize a first direction contour line extending in a first direction from the extracted contour line and a second direction contour line extending in a second direction different from the first direction, and recognize the road shape around the moving object based on the recognized multiple contour lines.

Images