Control device, mobile body, control system, method for operating control device, method for operating control system, program, and storage medium

Abstract

This control device for controlling a mobile body comprises: a travel route recognition means for recognizing a travel route; a tracking processing means for tracking the travel route; an instruction recognition means for recognizing a direction instruction of a user; and a control means for controlling movement of the mobile body on the basis of the recognition result of the travel route recognition means, the processing result of the tracking processing means, and the recognition result of the instruction recognition means.

Description

Control device, mobile body, control system, operation method of control device, operation method of control system, program, and storage medium

[0001] The present invention relates to a control device, a mobile body, a control system, an operation method of the control device, an operation method of the control system, a program, and a storage medium.

[0002] In recent years, small mobile bodies such as electric vehicles with about 1 to 2 passengers, called ultra-small mobility (also referred to as micromobility), and mobile robots that provide various services to people are known. Some of these mobile bodies perform autonomous driving while periodically generating a driving route to a destination. When generating a route, it is necessary to analyze a captured image by a camera or the like provided in the mobile body to recognize the road structure and the like, and generate a route according to the recognition result.

[0003] Patent Document 1 discloses that a trajectory when passing through a road shape such as an intersection having an entrance part and an exit part involving a route change is sequentially (stepwise) generated according to recognition information of the road shape obtained from a captured image.

[0004] Japanese Unexamined Patent Application Publication No. 2024-73949

[0005] However, in the technique described in Patent Document 1, since the traveling route of the mobile body is not tracked, it may be difficult to determine whether the traveling route after the route change is an appropriate traveling route that continues the traveling route before the route change.

[0006] The present invention has been made in view of the above problems, and an object thereof is to provide a technique for smoothly performing autonomous driving of a mobile body.

[0007] A control device according to an aspect of the present invention that achieves the above object is a control device that controls a mobile body, and includes a road recognition unit that recognizes a road, a tracking processing unit that tracks the road, an instruction recognition unit that recognizes a direction instruction of a user, and a control unit that controls the movement of the mobile body based on the recognition result of the road recognition unit, the processing result of the tracking processing unit, and the recognition result of the instruction recognition unit.

[0008] According to the present invention, it becomes possible to smoothly perform autonomous driving of a mobile object.

[0009] Other features and advantages of the present invention will become apparent from the following description with reference to the accompanying drawings. In the accompanying drawings, the same or similar components are given the same reference numeral.

[0010] The attached drawings are included in the specification and constitute a part thereof, illustrating embodiments of the present invention and are used to explain the principles of the present invention together with the description thereof. A diagram showing an example of the configuration of a mobile body according to one embodiment. A diagram showing an example of the configuration of a mobile body according to one embodiment. A diagram showing an example of the functional configuration of a control device that controls a mobile body according to one embodiment. A flowchart showing the procedure of processing performed by the control device according to one embodiment. A flowchart showing the procedure of processing performed by the control device according to one embodiment. An explanatory diagram of the extraction of recognition information according to one embodiment. An explanatory diagram of recognition information according to one embodiment, excluding the schematic diagram of the road. An explanatory diagram of the situation when turning right according to one embodiment. An explanatory diagram of path generation when turning right according to one embodiment. An explanatory diagram of the matching process according to one embodiment. An explanatory diagram when there is insufficient recognition information according to one embodiment.

[0011] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention as defined in the claims, and not all combinations of features described in the embodiments are essential to the invention. Two or more of the features described in the embodiments may be combined in any way. Furthermore, identical or similar configurations will be given the same reference numeral, and redundant descriptions will be omitted.

[0012] <Configuration of the Mobile Unit> Referring to Figures 1 and 2, the configuration of the mobile unit 1 according to this embodiment will be described. The mobile unit 1 is an ultra-compact mobility device that moves mainly by motor power, with a battery as the main power source. An ultra-compact mobility device is an ultra-compact vehicle that is more compact than a typical automobile and has a seating capacity of about one or two people. In this embodiment, a four-wheeled ultra-compact mobility device is described as an example of a mobile unit, but it may also be a three-wheeled vehicle or a saddle-type vehicle. Furthermore, the mobile unit is not limited to a vehicle, but may also be a mobile unit that carries luggage and runs alongside a person walking, or a mobile unit that leads a person. In addition, the mobile unit is not limited to a vehicle, but may also be a walking robot capable of autonomous movement. Furthermore, these ultra-compact mobility devices may be remotely controlled.

[0013] The mobile vehicle 1 is a four-wheeled vehicle equipped with a pair of left and right front wheels 3a and a pair of left and right rear wheels 3b. The mobile vehicle 1 is equipped with a seat 2 for one or two people. The control device 10 controls various operations of the mobile vehicle 1. The control device 10 comprises a CPU 101, a storage device 102, and a communication unit 103. The control operations of the control device 10 are realized by the CPU 101 reading and executing a computer program stored in the storage device 102. The CPU 101 may consist of one or more CPUs. The storage device 102 is one or more memories that store various information. For example, it stores information received from other devices and computer programs that are read and executed by the CPU 101. The communication unit 103 has the function of communicating with other devices via a network, either wired or wirelessly.

[0014] The external camera 20 is a camera that photographs the front of the mobile body 1. The internal camera 30 is a camera that photographs the inside of the mobile body 1. The drive mechanism 40 is a mechanism that rotates a pair of rear wheels 3b using a motor as the drive source. By rotating the pair of rear wheels 3b, the mobile body 1 can be moved forward or backward.

[0015] The braking unit 45 is a mechanism for performing brake operations, such as a disc brake. The steering unit 50 is a mechanism that changes the steering angle of a pair of front wheels 3a using a motor as the drive source. By changing the steering angle of the pair of front wheels 3a, the direction of travel of the mobile body 1 can be changed. The instruction input unit 60 is any device for instructing the direction of movement of the mobile body 1, and a multi-directional input device such as a joystick can be applied. However, it is not limited to a joystick, and may also be a turn signal, button, switch, etc. The driver can indicate their intention to turn right or left by operating the instruction input unit 60 before entering a road shape with an exit that requires a change of direction, such as an intersection. Furthermore, the mobile body 1 may be remotely controllable, and for example, a controller separate from the mobile body 1 may be equipped with a device that can input the direction of movement, such as a joystick. When a remote operator views the forward image of the mobile body 1 captured by the external camera 20 on a display (not shown) and operates the instruction input unit 60, the operation content is transmitted to the mobile body 1 via a communication device, allowing the remote operator to instruct the direction of movement of the mobile body 1. The display at this time may be provided on the mobile body 1, or a separate controller may have a display. In that case, the forward image data may be received from the mobile body 1 and displayed on the controller's display, which can then be viewed by the remote operator. Light Detection and Ranging (LID) devices 70 are positioned at the four corners of the mobile body 1 and can detect targets in the outside world.

[0016] The control device 10 uses the external camera 20 to capture images of the area in front of the moving vehicle 1 and extracts the road shape from the captured images. Then, it generates a route according to the recognition information indicating the extracted road shape and the operation instructions from the driver's instruction input unit 60. The recognition information is output by a machine learning model that processes the image information (captured images). The machine learning model performs calculations using a deep learning algorithm, for example, a deep neural network (DNN), to recognize the road shape contained in the image information. The recognition information may include information such as various road lines and lane information, the lane in which the vehicle is located (Ego lane), various intersections (Intersection), and various road entrances (Road entrance).

[0017] <Functional Configuration of the Control Device> Next, an example of the functional configuration of the control device according to this embodiment will be described with reference to Figure 3. The processing by each processing unit according to this embodiment is executed by the CPU 101. The control device 10 includes a path recognition unit 301, a tracking processing unit 302, an instruction recognition unit 303, a path determination unit 304, a determination unit 305, a matching processing unit 306, a path generation unit 307, and a control unit 308.

[0018] The lane recognition unit 301 recognizes the lane based on sensor data (for example, images captured by the external camera 20). The tracking processing unit 302 tracks the lane over time. The instruction recognition unit 303 recognizes user instructions input via the instruction input unit 60, such as a joystick. User instructions include, for example, a direction indication indicating the right or left turn the user wants to take.

[0019] The lane determination unit 304 provisionally selects a target lane (for example, a target lane such as the lane after turning right, the lane after turning left, or the lane after going straight at an intersection) from among one or more lanes recognized by the lane recognition unit 301, based on the content of the operation instruction recognized by the instruction recognition unit 303.

[0020] The determination unit 305 performs various determination processes. The matching processing unit 306 matches the track currently recognized by the track recognition unit 301 with a track previously recognized by the track recognition unit 301. If the matching is successful, it can be determined that the two are the same track; if the matching fails, it can be determined that the two are different tracks.

[0021] The path generation unit 307 generates a path (trajectory) from the current position of the moving body 1 through the intersection to the entrance of the lane ahead. For example, when turning right, a path is generated from the current position of the moving body 1 through the intersection to the lane after the right turn. The moving body 1 is controlled to move along this path. The control unit 308 performs various control processes, such as controlling the operation of the moving body 1 by controlling at least one of the drive unit 40, the braking unit 45, and the steering unit 50.

[0022] <Scene Description> Next, an example of an application scene of this embodiment will be described with reference to Figures 5 to 8 and Figure 10. In this embodiment, the explanation will be given using the case of driving through a crossroads, but the present invention is not limited to this and can also be applied to T-junctions, Y-junctions, intersections with five or more intersections, roundabouts, etc. First, the road recognition unit 301 extracts road recognition information as shown in Figure 5 from the captured image taken of the area in front of the moving body 1. The recognition information may include information such as road markings, center lines, lane information, the lane in which the vehicle is located (Ego lane), various intersections (Intersection), and entrances to various roads (Road entrance).

[0023] In Figure 5, 501 is the road lane marking, and 502 is the road's center line. 503 is the road entrance. Figure 6 is a diagram showing the various information detected in Figure 5 with the schematic road diagram removed. In Figure 6, 601 is the lane after a right turn, 602 is the lane after going straight, and 603 is the lane after a left turn. As shown in Figure 7, when a right turn direction indication is recognized, the lane 601 after the right turn is provisionally selected according to the flowcharts in Figures 4A and 4B described later, and after various processes, it is determined as the target route. Then, a path 701 is generated from the moving object 1 to the lane 601 after the right turn, and control is performed to move the moving object 1 according to that path 701. On the other hand, as shown in Figure 10, not all road shape and lane information is always successfully extracted as recognition information. In Figure 10, the lane marking, center line, road edge, and road entrance disappear on the left turn side. In such cases, the lane 603 after the left turn shown in Figure 6 will not be extracted. From here on, the control of mobile object 1 using this recognition information will be described in detail with reference to the flowchart.

[0024] <Processing> Figures 4A and 4B are flowcharts showing the processing steps performed by the control device according to this embodiment. The series of processes is executed repeatedly at predetermined intervals (e.g., 100 msec), i.e., at predetermined cycles. The series of processes is repeated each time an intersection is passed, but for example, an initialization process is performed before turning right or left at an intersection, and the series of processes is executed repeatedly until the intersection is passed. After the intersection is passed, the initialization process is performed before the next intersection.

[0025] In S401, the instruction recognition unit 303 acquires direction indication information based on the content of the operation instruction transmitted via the instruction input unit 60. For example, direction indication is given according to the tilt direction of the joystick. In S402, the track recognition unit 301 acquires sensor data. In this embodiment, the sensor is a camera, and the image of the area in front of the moving body 1 acquired by the outer camera 20 is acquired as sensor data.

[0026] In S403, the road recognition unit 301 extracts the road shape from the captured image based on the sensor data acquired in S402, and performs road recognition by deriving recognition information based on the extracted road shape.

[0027] In S404, the route determination unit 304 provisionally selects a target route (target lane) from one or more routes recognized by the route recognition unit 301, based on the content of the operation instruction recognized by the instruction recognition unit 303. For example, if the road shape is a cross-shaped intersection, as shown in Figure 6, one of the following is selected before entering the intersection: lane 601 after turning right, lane 603 after turning left, or lane 602 after going straight. For example, if a right turn direction instruction is input by the instruction input unit 60, lane 601 after turning right is provisionally selected as the target route. Similarly, if a left turn direction instruction is input by the instruction input unit 60, lane 603 after turning left is provisionally selected as the target route. If no right or left turn direction instruction is input by the instruction input unit 60, lane 602 after going straight is provisionally selected as the target route.

[0028] As shown in Figure 10, if the lane after the left turn is not properly detected, and only two lanes are detected—lane 601 after the right turn and lane 602 after going straight—and a left turn direction signal is input, the lane in the nearest direction, in this case lane 602 after going straight, will be provisionally selected.

[0029] In S405, the determination unit 305 determines whether there is an intention to turn right or left based on the content of the operation instruction received via the instruction input unit 60 recognized by the instruction recognition unit 303. If this step is Yes, proceed to S406. On the other hand, if this step is No, proceed to S431. Alternatively, if lost_counter < N (a predetermined value), proceed to S406. On the other hand, if lost_counter ≥ N (a predetermined value), proceed to S403. lost_counter is the value of a counter indicating the number of times the road has been lost.

[0030] Here, the series of processes shown in Figures 4A and 4B are repeatedly executed at predetermined intervals, and the tracking processing unit 302 constantly tracks the path of the moving object 1. In this process, it is determined whether the path recognized by the path recognition unit 301 is the same path as the path being tracked by the tracking processing unit 302 (i.e., the path recognized by the path recognition unit 301 in the previous process). If they are the same path, the path recognized by the path recognition unit 301 in this process is assigned the same ID (tracked_lane_id) and managed accordingly. On the other hand, if it is determined that they are not the same path, it is determined that the path has been lost, and lost_counter is incremented. In this step, a process is performed to determine whether the value of lost_counter is greater than N (a predetermined value).

[0031] In S406, the determination unit 305 determines whether a target track (target lane) exists, or more specifically, whether the target track (target lane) has been detected with sufficient accuracy to be used for path generation (trajectory generation) as described later. For example, if lanes with a width between the lane markings (more specifically, the distance between the left lane marking and the center line) of a predetermined width or more have been detected along the direction in which the lane markings extend for a length of a predetermined value or more, it may be determined that detection has been achieved with sufficient accuracy. In the provisional selection in S404, provisional selection is performed even if detection has not been achieved with sufficient accuracy, but in this step, it is determined whether detection has been achieved with sufficient accuracy, that is, whether the provisionally selected target track (lane) is valid. If a target track (target lane) exists, the attribute tracked_lane_id is assigned to the target track (target lane). For example, the system may be configured to calculate the similarity between the position and / or size of the target track (target lane) in the forward image detected in the previous process and the position and / or size of the target track (target lane) in the forward image detected in the current process, and assign a different tracked_lane_id than the previous one if the similarity is below a threshold. If this step is Yes, the system proceeds to S407. On the other hand, if this step is No, the system proceeds to S428.

[0032] In S407, the determination unit 305 determines whether a target track (target lane) has existed in the past. That is, for a series of processes performed at a predetermined cycle, it determines whether a valid target track was detected in the previous process. This corresponds to determining whether lane_detected is True as a result of the previous process. Here, lane_detected is a flag that is True if a target track (target lane) was detected and False if it was not detected. It is also False if this flowchart process is performed for the first time after initialization. If this step is Yes, the process proceeds to S408. On the other hand, if this step is No, the process proceeds to S424.

[0033] In S408, the determination unit 305 determines whether the tracking of the target track was successful. That is, the determination unit 305 determines whether the track recognized by the track recognition unit 301 in this process is the same track as the track being tracked by the tracking processing unit 302. This step can be performed by determining whether there has been a change in the value of tracked_lane_id. For example, if the ID of the track being tracked by the tracking processing unit 302 is 1001, and the ID of the track recognized by the track recognition unit 301 in this process is 1002, it means that they have been recognized as different tracks. In other words, the determination unit 305 determines whether the tracked_lane_id assigned in this process matches the tracked_lane_id held after the previous process. In reality, even if they are the same track, it is possible that they may be determined to be different tracks in the recognition process. If this step is Yes, the process proceeds to S409. On the other hand, if this step is No, proceed to S414.

[0034] In S409, the path generation unit 307 generates a path (trajectory of the moving object 1) to the target path detected in the current process. In S410, the control unit 308 sets lane_detected to True. In S411, the control unit 308 sets lost_counter to 0. In S412, the control unit 308 updates tracked_lane_id, which has been held since the previous process.

[0035] If the process proceeds to S412 in Figure 4B after selecting Yes in S408, tracked_lane_id will be set to the same value as before because tracking was successful. If the process proceeds to S412 after selecting No in S416, it may appear at first glance that tracking failed, but as a result of the matching process described later, it will be found that tracking did not fail, so the value will be set to the same value as before. If the process proceeds to S412 after selecting Yes in S424, the target track has been detected for the first time, so tracked_lane_id will be newly assigned.

[0036] In S413, the control unit 308 outputs a path and controls the movement of the mobile body 1 along the path.

[0037] In S414, the determination unit 305 determines whether there has been a change in the intention to turn right or left, based on the content of the operation instruction received via the instruction input unit 60, which has been recognized by the instruction recognition unit 303. This process is, for example, a process to check whether there has been any change in the right turn direction instruction after inputting a right turn direction instruction in S404 to S405. If this step is Yes, the process proceeds to S415. On the other hand, if this step is No, the process proceeds to S420.

[0038] In S415, the matching processing unit 306 matches the track recognized by the track recognition unit 301 with a track previously recognized by the track recognition unit. For example, it extracts the lane markings (white lines) of the track (lane) recognized this time and extracts the lane markings (white lines) of the previously recognized track (lane). Then, it calculates the distance between the left and right lane markings in the direction of travel of the track (lane) at predetermined intervals (e.g., 1m intervals), calculates the mean square error of these distances, and determines that the two are the same track if the calculated value is less than or equal to a threshold (matching successful). On the other hand, if the calculated value exceeds the threshold, it determines that the two are different tracks (matching failed).

[0039] For example, as shown in Figure 9, for the lane after a right turn, the distances between the previously recognized left lane marking 901 and the currently recognized left lane marking 911 are calculated at predetermined intervals, and these values ​​are D1, D2, D3, ..., D6. The mean square error of these values ​​is then calculated. Similarly, for the lane after a right turn, the distances between the previously recognized right lane marking 902 and the currently recognized right lane marking 912 are calculated at predetermined intervals, and the mean square error of these values ​​is calculated. Furthermore, the angle at which the previous lane and the current lane intersect may be calculated, and a condition may be added that the calculated value must be less than or equal to a threshold.

[0040] In S416, the determination unit 305 determines whether the matching by the matching processing unit 306 was successful. Successful matching means that although the tracking process failed and it appeared to be a different target track, it is actually considered to be the same target track. If this step is Yes, proceed to S409. On the other hand, if this step is No, proceed to S417.

[0041] In S417, the control unit 308 decides to use the path generated during the previous processing. In S418, the control unit 308 sets lane_detected to True. In S419, the control unit 308 increments lost_counter. Then proceed to S413.

[0042] In S420, the path generation unit 307 generates a path (the trajectory of the moving body 1) to the target runway detected in the current process. In S421, the control unit 308 sets lane_detected to True. In S422, the control unit 308 sets lost_counter to 0.

[0043] In S423, the control unit 308 updates the tracked_lane_id held through the previous process. Here, since a direction instruction different from that at the time of the previous process is input, the target lane is different from the previous one. For example, when a right direction instruction was input and the lane after a right turn was detected during the previous process, and a left direction instruction is input and the lane after a left turn is selected in the current process, this corresponds to this case. In this case, since the target lane to be tracked is reset, after passing through No in S408, the process proceeds to the processes of S420 to S423. Therefore, the tracked_lane_id held through the previous process is changed to a different ID and updated. Then, the process proceeds to S413.

[0044] In S424, the determination unit 305 determines whether the direction indicated by the direction instruction and the direction in which the target lane is located are the same direction based on the content of the operation instruction via the instruction input unit 60 recognized by the instruction recognition unit 303. If this step is Yes, the process proceeds to S409. On the other hand, if this step is No, the process proceeds to S425.

[0045] In S425, the path generation unit 307 generates a path (the trajectory of the moving body 1) to the target runway detected in the current process. In S426, the control unit 308 sets lane_detected to False. In S427, the control unit 308 sets lost_counter to 0. Then, the process proceeds to S413.

[0046] In S428, the control unit 308 determines to use the path generated during the previous process. Here, since the target lane has not been detected, the path used in the previous process is utilized. In S429, the control unit 308 sets lane_detected to False. In S430, the control unit 308 increments lost_counter. Then, it proceeds to S413.

[0047] In S431, the determination unit 305 determines whether a target lane (target path) exists, or more specifically, whether the target lane (target path) has been detected with sufficient accuracy to be used for path generation (trajectory generation) described later. This step is the same process as the process in S406. If this step is Yes, it proceeds to S432. On the other hand, if this step is No, it proceeds to S436.

[0048] In S432, the path generation unit 307 generates a path (trajectory of the moving body 1) to the target lane detected in this process. Here, since there is no intention to turn right or left, a path is generated to the lane closest to the straight-ahead direction among the detected lanes. For example, if the lane after straight-ahead is detected as the target lane, a path to that lane after straight-ahead is generated.

[0049] In S433, the control unit 308 sets lane_detected to True. Here, since the lane after straight-ahead has been detected, it is set to True. In S434, the control unit 308 sets lost_counter to 0. In S435, the control unit 308 updates the tracked_lane_id held through the previous process. Then, it proceeds to S413.

[0050] In S436, the control unit 308 executes exception handling. In this case, for example, although there is no intention to turn right or left and straight-ahead is assumed, no targetable lane has been detected, so a process of decelerating or stopping the moving body 1 is performed. The above is the series of processes in FIGS. 4A and 4B, which are repeatedly executed until the intersection passage is completed.

[0051] As described above, according to this embodiment, by tracking the path of the moving object, it becomes possible to enable the autonomous driving of the moving object to proceed smoothly.

[0052] <Example 1> For example, suppose a moving object 1 is about to enter an intersection, a right turn direction signal is input, and after going through S401 to S405, the result is Yes at S405 and proceeds to S406. If the lane after the right turn is detected with sufficient accuracy to be used for path generation, the result is Yes at S406 and proceeds to S407. If that lane after the right turn has been detected as a target road (target lane) in the past, the result is Yes at S407 and proceeds to S408. At S408, suppose that the tracked_lane_id is different between the past lane and the current lane, i.e., tracking has failed. In that case, proceed to S414, but if there is no change in the intention to turn right or left, i.e., if the right turn direction signal is still input during this process as in the previous process, proceed to S415 and S416.

[0053] If tracking fails but matching is successful, it simply means that the recognition process has determined that it is a different lane, so the process proceeds to S409 and the same processing as when tracking was successful is performed. Also, if matching fails in S416, it is possible that the lane detected this time is a different lane from the previous one that was mistakenly detected, so in S417 a previously generated path (a path generated in the previous process) is used. Then, in S418 lane_detected = True and in S419 lost_counter is incremented.

[0054] On the other hand, if in S414 a left turn signal was input in the previous process and a right turn signal is input in the current process, the process proceeds to S420. In that case, in S420 a path is generated toward the lane after the right turn, lane_detected = True is set in S421, lost_counter is set to 0 in S422, and tracked_lane_id is updated in S423.

[0055] <Example 2> Similarly, suppose that moving object 1 is about to enter an intersection, a right turn direction signal is input, and after proceeding through S401 to S405, the response is Yes at S405 and proceeds to S406. If the lane after the right turn is detected with sufficient accuracy to be used for path generation, the response is Yes at S406 and proceeds to S407. If the lane after the right turn has not been detected as a target route (target lane) in the past, the response is No at S407 and proceeds to S424.

[0056] Here, let's assume that in S404, when the target lane is tentatively selected, the direction indicator is for a right turn, but the lane after the right turn and the lane after going straight have not been detected, and the lane after the left turn is tentatively selected. In such a case, in S424, it is determined that the direction indicator (right) and the target lane (left) are not in the same direction, and the process proceeds to S425. In that case, a path to the lane after the left turn is generated in S425, lane_detected is set to False in S426, and lost_counter is set to 0 in S427.

[0057] On the other hand, when the target lane is tentatively selected in S404, the direction indicator is for a right turn, the lane after the right turn is detected, and the lane after the right turn is tentatively selected. In such a case, in S424, it is determined that the direction indicator (right) and the target lane (right) are in the same direction, and the process proceeds to S409. In that case, a path to the lane after the right turn is generated in S409, lane_detected is set to True in S410, and lost_counter is set to 0 in S411. Also, since the target lane has not been detected in the past, a new tracked_lane_id is assigned in S412.

[0058] <Case 3> Assume that moving object 1 is about to enter an intersection, and a right turn direction signal is input, proceeding through S401 to S405, and then Yes at S405, and proceeding to S406. If the lane after the right turn is not detected with sufficient accuracy to be used for path generation, then No at S406, and proceeding to S428. In this case, a new path cannot be generated, so the path generated in the previous process is used. Then, lane_detected is set to False at S429, and lost_counter is incremented at S430.

[0059] <Case 4> If the moving object 1 is about to enter an intersection and no right or left turn signal has been input, the lane after going straight is tentatively selected in S404. In that case, the result is No in S405, and the process proceeds to S431. If the lane after going straight is detected with sufficient accuracy to be used for path generation, the result is Yes in S431, and the process proceeds to S432. In S432, a path to the lane after going straight is generated, lane_detected is set to True in S433, and lost_counter is set to 0 in S434. Then, tracked_lane_id is updated in S435. On the other hand, if the lane after going straight is not detected with sufficient accuracy to be used for path generation, the result is No in S431, and the process proceeds to S436. In S436, exception processing is performed to decelerate or stop the moving object 1.

[0060] <Example 5> When the value of the counter indicating the number of times the track has been lost is greater than or equal to a threshold (i.e., lost_counter ≥ N) (No in S405), if the track recognition unit 301 is unable to recognize the track (No in 431), the moving body 1 may be decelerated or stopped (S436).

[0061] <Modification> In the above embodiment, the example mainly described was when the control device 10 performs the processing, but it is not limited to this example. A part of the processing performed by the control device 10 may be performed by a controller, server device, etc., separate from the control device 10 (or mobile body 1). For example, the processing of the instruction recognition unit 303 among the path recognition unit 301, tracking processing unit 302, instruction recognition unit 303, and control unit 308 may be performed by a separate controller (equipped with a joystick, etc.) that communicates with the control device 10, and the instruction recognition result may be transmitted to the control device 10, and the control device 10 may use the received instruction recognition result. In other words, instead of the control device 10 alone performing all the processing, the processing according to the above embodiment may be shared and performed by the entire control system, which may include multiple devices.

[0062] <Summary of Embodiments> 1. The control device (10) according to the above embodiment is a control device for controlling a moving body (1), and comprises: a path recognition means (301) for recognizing a path; a tracking processing means (302) for tracking the path; an instruction recognition means (303) for recognizing a user's direction indication; and a control means (308) for controlling the movement of the moving body based on the recognition result of the path recognition means, the processing result of the tracking processing means, and the recognition result of the instruction recognition means.

[0063] According to this embodiment, tracking the path of the moving object makes it possible to enable the moving object to navigate autonomously smoothly.

[0064] 2. The control device (10) according to the above embodiment further comprises: determination means (305) for determining whether the track recognized by the track recognition means is the same as the track being tracked by the tracking processing means; and matching processing means (306) for matching the track recognized by the track recognition means with a track previously recognized by the track recognition means if the determination means determines that they are not the same; and the control means controls the movement of the moving body based on the processing result of the matching processing means and the recognition result of the instruction recognition means.

[0065] According to this embodiment, even if tracking fails, the movement of the moving object can be appropriately controlled.

[0066] 3. In the control device (10) according to the above embodiment, if the matching by the matching processing means is successful, the control means controls the movement of the moving body toward a target path that is located in the direction indicated by the user and is recognized by the path recognition means.

[0067] According to this embodiment, even if tracking fails, if matching is successful and the same target path as the previous target path is actually detected, the moving object can be appropriately moved toward that target path.

[0068] 4. The control device (10) according to the above embodiment further comprises a path generation means (307) for generating a path to the target path, and the control means moves the moving body along the path.

[0069] According to this embodiment, it is possible to generate a smooth path from the current position of the moving object towards the target path, passing through intersections and the like, and to move the moving object appropriately.

[0070] 5. The control device (10) according to the above embodiment further comprises a path generation means (307) that generates a path to a target path that is located in the direction indicated by the user and is recognized by the path recognition means, and the control means moves the moving body using a path previously generated by the path generation means if matching by the matching processing means fails.

[0071] According to this embodiment, if tracking fails and matching also fails, the reliability of the target path in the current process is low, so a path is not generated, and the moving object can be controlled using the reliable path generated in the previous process. Therefore, if the target path in the current process is detected in an unexpected position, it is expected that the behavior of the moving object will not be smooth, but such unnatural behavior can be prevented.

[0072] 6. The control device (10) according to the above embodiment further comprises a second determination means (305) that, when the determination means determines that it is not the same road, determines whether the content of the user's direction indication recognized by the instruction recognition means has not changed from the content of the user's direction indication previously recognized by the instruction recognition means, and the matching processing means executes a matching process when the second determination means determines that there has been no change.

[0073] If a different direction instruction is input than in the previous processing, it is clear that the target lane detected last time and the target lane detected this time are different, and therefore matching is not necessary. According to this embodiment, unnecessary matching processing can be suppressed, thereby reducing the processing load.

[0074] 7. In the control device (10) according to the above embodiment, the control means slows down or stops the moving body when the value of the counter indicating the number of times the track has been lost is greater than or equal to a threshold and the track recognition means is unable to recognize the track.

[0075] According to this embodiment, if the track cannot be detected for an extended period, it is possible that the track does not actually exist or that there is a problem with equipment such as a camera. In such cases, safety can be improved by slowing down or stopping the moving object.

[0076] 8. In the control device (10) according to the above embodiment, the control means increments the value of the counter if the track cannot be recognized by the track recognition means.

[0077] According to this embodiment, the number of times the track is lost can be appropriately managed, and this information can be used for other processing.

[0078] 9. The control device (10) according to the above embodiment further comprises: determination means (305) for determining whether the track recognized by the track recognition means is the same as the track being tracked by the tracking processing means; and matching processing means (306) for matching the track recognized by the track recognition means with a track previously recognized by the track recognition means if the determination means determines that they are not the same; and the control means increments the value of the counter if the matching by the matching processing means fails.

[0079] According to this embodiment, if tracking fails and matching also fails, it is considered that the previous target track and the current target track are different tracks, and this is counted as a track loss. This allows for proper management of the number of track losses, and the information can be used for other processes.

[0080] 10. In the control device (10) according to the above embodiment, the processing of the control device is repeatedly executed at predetermined intervals.

[0081] According to this embodiment, the results of the previous or earlier processing can be utilized in the current processing.

[0082] 11. In the control device (10) according to the above embodiment, the instruction recognition means recognizes the user's direction indication based on information input via a joystick or button.

[0083] According to this embodiment, the user, who is the occupant of the mobile body 1, can input a direction of travel in their desired direction with simple operation.

[0084] 12. In the control device (10) according to the above embodiment, the user's direction indication includes an instruction to turn right or turn left.

[0085] This allows the user, who is an occupant of the mobile vehicle 1, to input a right or left turn signal with simple operation.

[0086] 13. In the control device (10) according to the above embodiment, the track recognition means recognizes the track based on the information acquired by the sensor.

[0087] According to this embodiment, the road can be recognized from sensor information.

[0088] 14. In the control device (10) according to the above embodiment, the sensor is a camera (20), and the information acquired by the sensor is a captured image acquired by the camera.

[0089] According to this embodiment, it is possible to recognize the road from the captured image.

[0090] 15. The mobile body (1) according to the above embodiment is a mobile body equipped with the control device according to the above embodiment.

[0091] According to this embodiment, the functions of the control device according to the above embodiment can be realized in a mobile device.

[0092] 16. The control system according to the above embodiment is a control system for controlling a moving body (1), comprising: a path recognition means for recognizing a path; a tracking processing means for tracking the path; an instruction recognition means for recognizing a user's direction indication; and a control means for controlling the movement of the moving body based on the recognition result of the path recognition means, the processing result of the tracking processing means, and the recognition result of the instruction recognition means.

[0093] According to this embodiment, tracking the path of the moving object makes it possible to enable the moving object to navigate autonomously smoothly.

[0094] 17. The operation method of the control device (10) according to the above embodiment is an operation method of a control device that controls a mobile body (1), comprising: a path recognition step of recognizing a path; a tracking processing step of tracking the path; an instruction recognition step of recognizing a user's direction indication; and a control step of controlling the movement of the mobile body based on the recognition result in the path recognition step, the processing result in the tracking processing step, and the recognition result in the instruction recognition step.

[0095] According to this embodiment, tracking the path of the moving object makes it possible to enable the moving object to navigate autonomously smoothly.

[0096] 18. The method of operation of the system according to the above embodiment is a method of operation of a control system for controlling a moving body (1), comprising: a path recognition step of recognizing a path; a tracking processing step of tracking the path; an instruction recognition step of recognizing a user's direction indication; and a control step of controlling the movement of the moving body based on the recognition result in the path recognition step, the processing result in the tracking processing step, and the recognition result in the instruction recognition step.

[0097] According to this embodiment, tracking the path of the moving object makes it possible to enable the moving object to navigate autonomously smoothly.

[0098] 19. The program according to the above embodiment is a program that causes a computer to execute the operation method of the control device according to the above embodiment.

[0099] According to this embodiment, the operation method of the control device can be realized as a program.

[0100] 20. The storage medium according to the above embodiment is a storage medium in which a program is stored that causes a computer to execute the operation method of the control device according to the above embodiment.

[0101] According to this embodiment, the operation method of the control device can be realized as a storage medium.

[0102] (Other Embodiments) In addition, a program that implements one or more functions described in each embodiment is supplied to a system or device via a network or storage medium, and one or more processors in the computer of the system or device can read and execute this program. The present invention can also be realized in such a form.

[0103] The invention is not limited to the embodiments described above, and various modifications and changes are possible within the scope of the gist of the invention.

[0104] 1: Mobile unit, 10: Control device, 20: External camera, 301: Path recognition unit, 302: Tracking processing unit, 303: Instruction recognition unit, 304: Path determination unit, 305: Judgment unit, 306: Matching processing unit, 307: Path generation unit, 308: Control unit

Claims

1. A control device for controlling a moving object, comprising: a path recognition means for recognizing a path; a tracking processing means for tracking the path; an instruction recognition means for recognizing a user's direction indication; and a control means for controlling the movement of the moving object based on the recognition result of the path recognition means, the processing result of the tracking processing means, and the recognition result of the instruction recognition means.

2. The control device according to claim 1, further comprising: determination means for determining whether the track recognized by the track recognition means is the same as the track being tracked by the tracking processing means; and matching processing means for matching the track recognized by the track recognition means with a track previously recognized by the track recognition means if the determination means determines that they are not the same, wherein the control means controls the movement of the moving body based on the processing result of the matching processing means and the recognition result of the instruction recognition means.

3. The control device according to claim 2, characterized in that, when matching by the matching processing means is successful, the control means controls the movement of the moving body toward a target path which is located in the direction indicated by the user and is recognized by the path recognition means.

4. The control device according to claim 3, further comprising path generation means for generating a path to the target track, wherein the control means moves the moving body along the path.

5. The control device according to claim 2, further comprising path generation means for generating a path to a target path that is located in the direction indicated by the user and is recognized by the path recognition means, wherein the control means moves the moving body using a path previously generated by the path generation means if matching by the matching processing means fails.

6. The control device according to claim 2, further comprising a second determination means for determining whether the content of the user's direction indication recognized by the instruction recognition means has not changed from the content of the user's direction indication previously recognized by the instruction recognition means, when the determination means determines that it is not the same road, wherein the matching processing means executes a matching process when the second determination means determines that there has been no change.

7. The control device according to claim 1, characterized in that the control means slows down or stops the moving body when the value of a counter indicating the number of times the track has been lost is above a threshold and the track recognition means is unable to recognize the track.

8. The control device according to claim 7, characterized in that the control means increments the value of the counter if the track cannot be recognized by the track recognition means.

9. The control device according to claim 7, further comprising: determination means for determining whether the track recognized by the track recognition means is the same as the track being tracked by the tracking processing means; and matching processing means for matching the track recognized by the track recognition means with a track previously recognized by the track recognition means if the determination means determines that they are not the same, wherein the control means increments the value of the counter if the matching by the matching processing means fails.

10. The control device according to claim 1, characterized in that the processing of the control device is repeatedly performed at predetermined intervals.

11. The control device according to claim 1, characterized in that the instruction recognition means recognizes the user's direction indication based on information input via a joystick or button.

12. The control device according to claim 1, characterized in that the user's direction indication includes an indication to turn right or turn left.

13. The control device according to claim 1, characterized in that the track recognition means recognizes a track based on information acquired by a sensor.

14. The control device according to claim 13, characterized in that the sensor is a camera, and the information acquired by the sensor is a captured image acquired by the camera.

15. A mobile body characterized by comprising the control device described in claim 1.

16. A control system for controlling a moving object, comprising: a path recognition means for recognizing a path; a tracking processing means for tracking the path; an instruction recognition means for recognizing a user's direction indication; and a control means for controlling the movement of the moving object based on the recognition result of the path recognition means, the processing result of the tracking processing means, and the recognition result of the instruction recognition means.

17. A method for operating a control device for controlling a moving object, comprising: a path recognition step of recognizing a path; a tracking processing step of tracking the path; an instruction recognition step of recognizing a user's direction indication; and a control step of controlling the movement of the moving object based on the recognition result in the path recognition step, the processing result in the tracking processing step, and the recognition result in the instruction recognition step.

18. A method for operating a control system for controlling a moving object, comprising: a path recognition step of recognizing a path; a tracking processing step of tracking the path; an instruction recognition step of recognizing a user's direction indication; and a control step of controlling the movement of the moving object based on the recognition result in the path recognition step, the processing result in the tracking processing step, and the recognition result in the instruction recognition step.

19. A program for causing a computer to execute the operation method of the control device described in claim 17.

20. A storage medium storing a program for causing a computer to execute the operation method of the control device described in claim 17.

Images