Vehicle control method and vehicle control device
The vehicle control method and device enable seamless transitions from autonomous to remote parking, addressing the lack of integrated parking assistance by using sensors and cameras to ensure accurate vehicle positioning and control.
Patent Information
- Application Number
- PCT/JP2026/000373
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-05-15
- Filing Date
- 2026-01-08
- Publication Date
- 2026-11-19
AI Technical Summary
Existing vehicle parking systems lack seamless transition from autonomous driving to remote parking, often resulting in inadequate parking assistance.
A vehicle control method and device that integrates autonomous driving, remote parking, and manual operation, utilizing sensors, cameras, and a display to facilitate smooth transitions and accurate parking through a combination of autonomous and remote control modes.
Provides effective parking assistance by ensuring vehicles can autonomously or remotely park accurately, enhancing user convenience and safety in challenging parking environments.
Smart Images

Figure JP2026000373_19112026_PF_FP_ABST
Abstract
Description
Vehicle control method and vehicle control device
[0001] The present disclosure relates to a vehicle control method and a vehicle control device.
[0002] In general, parking spaces in homes often have many narrow locations, which can make parking difficult, so there is a high demand for parking assistance. Accordingly, systems that park a vehicle via autonomous driving, and systems that use remote parking to park a vehicle via remote operation from outside the vehicle using a smartphone or the like have been disclosed.
[0003] However, in the prior art, no consideration has been given to the transition from autonomous driving to remote parking, and it has sometimes been difficult to perform appropriate parking assistance.
[0004] Japanese Patent No. 7272399
[0005] An object of the present disclosure is to provide a vehicle control method and a vehicle control device that can provide appropriate parking assistance.
[0006] The vehicle control method according to this disclosure is a vehicle control method executed by a vehicle control device mounted on a vehicle that includes an operating device that receives at least a brake operation and an operation to specify forward, reverse, or park from an occupant, a camera that acquires surrounding images, a display device that can be viewed by the occupant, and a movement control device that controls at least acceleration, deceleration, and steering. The vehicle control method according to this disclosure, when the operating device receives an automatic parking start request, starts automatic parking control that causes the movement control device to control at least acceleration, deceleration, and steering based on a predetermined route, causing the vehicle to autonomously drive to a target parking position. The vehicle control method according to this disclosure displays on the display device that the vehicle will switch to remote parking mode when the vehicle stops due to the brake operation of the operating device and the operating device receives an operation to specify parking during the autonomous driving of the vehicle. The vehicle control method according to this disclosure switches to remote parking mode when the vehicle stops due to the brake operation of the operating device and the operating device receives an operation to specify parking during the autonomous driving of the vehicle. The vehicle control method according to this disclosure, in the event that the vehicle does not stop due to at least the brake operation of the operating device while the vehicle is autonomously driving, causes the movement control device to control at least acceleration / deceleration and steering based on the predetermined path, thereby parking the vehicle autonomously to the target parking position.
[0007] The vehicle control method and vehicle control device described herein can provide suitable parking assistance.
[0008] Figure 1 is a block diagram showing an example of the overall configuration of the parking assistance system of the embodiment. Figure 2 is an explanatory diagram of an example of the arrangement of sensors and cameras. Figure 3 is a schematic diagram showing an example of the configuration of the exterior of a vehicle. Figure 4 is a diagram showing an example of the configuration of the area near the driver's seat of the vehicle of the embodiment. Figure 5 is an explanatory diagram of an example of parking assistance of the embodiment. Figure 6 is a schematic diagram of an example of a screen. Figure 7A is a schematic diagram of an example of a screen. Figure 7B is a schematic diagram of an example of a screen. Figure 7C is a schematic diagram of an example of a screen. Figure 7D is a schematic diagram of an example of a screen. Figure 8 is a flowchart showing an example of the information processing flow performed by the vehicle control device of the embodiment. Figure 9 is a block diagram showing an example of a hardware configuration.
[0009] Hereinafter, embodiments of the vehicle control method and vehicle control device according to this disclosure will be described with reference to the drawings. In the following description of embodiments, the same parts will be denoted by the same reference numerals, and redundant descriptions will be omitted.
[0010] Figure 1 is a block diagram showing an example of the overall configuration of the parking assistance system 1000 of this embodiment.
[0011] The parking assistance system 1000 comprises a vehicle 1 and a wireless communication terminal 3. The vehicle 1 and the wireless communication terminal 3 are configured to communicate wirelessly with each other.
[0012] The wireless communication terminal 3 is an information processing device that can communicate wirelessly with the vehicle 1. The wireless communication terminal 3 is an information processing device that receives commands from the user for remote operation of the vehicle 1. The wireless communication terminal 3 is, for example, a mobile device such as a smartphone.
[0013] The wireless communication terminal 3 comprises a control unit 30, a communication unit 32, a storage unit 34, and a UI (user interface) unit 36. The communication unit 32, the storage unit 34, the UI unit 36, and the control unit 30 are communicated with each other.
[0014] The communication unit 32 is a communication function unit that wirelessly communicates with external devices such as the vehicle control device 10, which will be described later, and is installed on the vehicle 1. The storage unit 34 stores various types of data. The UI unit 36 receives output and input from the user. The UI unit 36 has a display function and an input function. The display function is a function that displays various types of information. The display function is realized by, for example, a display, a projection device, etc. The input function receives various instructions from the user. The input function is realized by, for example, an operation panel, a keyboard, a mouse or other pointing device, etc. In this embodiment, a configuration in which the UI unit 36 is a touch panel with a display and an operation panel superimposed on each other will be described as an example.
[0015] The control unit 30 performs information processing on the wireless communication terminal 3. In this embodiment, the control unit 30 has a remote operation application 30A pre-installed on it.
[0016] The remote control application 30A is application software for enabling remote control of the vehicle 1. Remote control of the vehicle 1 means remotely controlling the vehicle 1 through user operations on the wireless communication terminal 3.
[0017] The remote operation application 30A establishes a link with the vehicle control device 10, and when the link is established, it receives a predetermined operation from the user via the UI unit 36, and while it is receiving the predetermined operation, it sends a remote operation instruction received signal to the vehicle control device 12 to indicate that a remote operation instruction is being received.
[0018] Establishing a link means that the remote control application 30A and the vehicle control device 10 are connected in a way that enables wireless communication. The remote control application 30A and the vehicle control device 10 each perform a process to attempt to establish a link by known method so that they can communicate wirelessly with each other in accordance with a predetermined communication standard. When a link is established between the remote control application 30A and the vehicle control device 10, the remote control application 30A and the vehicle control device 10 are connected in a way that enables wireless communication.
[0019] Details of the remote operation of vehicle 1 using wireless communication terminal 3 will be described later.
[0020] Vehicle 1 comprises a vehicle control device 10, a movement control device 12, a sensor 14, a camera 16, a storage device 18, an operating device 20, a display device 22, and a wireless communication device 24.
[0021] The vehicle control device 10 is connected to the movement control device 12, sensor 14, camera 16, storage device 18, operating device 20, display device 22, and wireless communication device 24 so that data or signals can be exchanged. In other words, the vehicle control device 10 is configured to be communicatively connected to at least the movement control device 12, camera 16, operating device 20, display device 22, and wireless communication device 24.
[0022] The motion control device 12 controls at least the acceleration, deceleration, and steering of the vehicle 1. The motion control device 12 is a means for realizing the driving, braking, and turning movements necessary for the vehicle 1 to travel. For example, the motion control device 12 is composed of a drive motor, a power transmission mechanism, a brake device, a steering device, etc., and an electronic vehicle control device that controls them. The motion control device 12 drives the vehicle 1 by, for example, generating power with a drive motor and transmitting power to the wheels via a power transmission mechanism. The power transmission mechanism is, for example, a propeller shaft, a differential gear, and a drive shaft.
[0023] To control steering at least means that the motion control device 12 controls at least one of the driving, braking, and turning motions necessary for the vehicle 1 to travel. In other words, to control steering means that the motion control device 12 controls at least one of the following: turning direction by steering, vehicle speed and acceleration by accelerator steering, and deceleration and stopping by brake steering. To control acceleration and deceleration at least means that the motion control device 12 controls at least one of the acceleration and deceleration of the vehicle 1.
[0024] The movement control device 12 includes, in detail, an auxiliary control device 12A, a brake control device 12B, an engine control device 12C, and a power steering control device 12D. The brake control device 12B, the engine control device 12C, and the power steering control device 12D can be collectively referred to as an actuator control unit that controls the operation of the vehicle 1.
[0025] The auxiliary control device 12A is a control device that monitors the transmission status of the vehicle control device 10 and operates to perform appropriate degraded control as a backup in the event that the vehicle control device 10 fails. However, if safety can be ensured by having a degraded control function within the vehicle control device 10 even in the event that the vehicle control device 10 fails, then degraded control is unnecessary.
[0026] The brake control device 12B is a control device that controls the brakes of the vehicle 1. Brake control is sometimes referred to as braking force control. For example, the brake control device 12B controls the brakes of the vehicle 1 in response to the strengthening and weakening of the brake pedal operation by the occupant. Strengthening the brake pedal operation by the occupant specifically means pressing the brake pedal down by the occupant. In addition, the brake control device 12B controls the brakes in response to the surrounding conditions acquired during autonomous driving.
[0027] The engine control device 12C is a control device that controls the engine that generates the driving force of the vehicle 1. The power steering control device 12D is a control device that controls the power steering of the vehicle 1.
[0028] Sensor 14 is mounted on the vehicle 1 and acquires at least the external conditions of the vehicle 1. In detail, sensor 14 is a variety of sensors that detect the driving state of the vehicle 1 and the external conditions of the vehicle 1. Sensor 14 includes, for example, an accelerator position sensor that detects the accelerator position, a steering angle sensor that detects the steering angle of the steering device, an acceleration sensor that detects the acceleration acting on the vehicle 1 in the longitudinal direction, a torque sensor that detects the torque acting on the power transmission mechanism between the wheels and the drive motor of the vehicle 1, a vehicle speed sensor that detects the vehicle speed of the vehicle 1, a wheel speed sensor, GPS (Global Positioning System), etc. Sensor 14 also includes LiDAR (Light detection and ranging), radar, or ultrasonic sensors, etc. Sensor 14 outputs the sensor information obtained through detection to the vehicle control device 10.
[0029] Camera 16 is a surrounding sensor mounted on vehicle 1 that monitors the environment around vehicle 1. Camera 16 has an image sensor. In this embodiment, camera 16 photographs the area around vehicle 1 and outputs the surrounding video data obtained from the photography to vehicle control device 10. The surrounding video data consists of multiple captured image data taken in chronological order around vehicle 1. Hereinafter, the surrounding video data may be simply referred to as surrounding video. In this embodiment, camera 16 is also used to detect objects present around vehicle 1 and to estimate the location of vehicle 1 based on the positional relationship between vehicle 1 and the objects present around vehicle 1.
[0030] The position, number, and shooting direction of the cameras 16 are pre-adjusted so that they can photograph the area around the vehicle 1. For example, the vehicle 1 is equipped with four cameras 16 arranged to photograph the front, rear, left, and right directions of the vehicle 1. However, the number of cameras 16 installed on the vehicle 1 is not limited to four.
[0031] The storage device 18 stores various types of data. In this embodiment, for example, route data 18A and map data 18B are stored in the storage device 18. Details of the route data 18A and map data 18B will be described later.
[0032] The storage device 18 is, for example, an auxiliary storage device such as an HDD (Hard Disk Drive), SSD (Solid State Drive), or flash memory. At least a portion of the data contained in the storage device 18 may be stored in an external storage device such as a server device located outside the vehicle 1 and connected to the vehicle control device 10 in a communicative manner.
[0033] The operating device 20 receives operations from the occupant of the vehicle 1. In this embodiment, the operating device 20 receives at least the occupant's brake operation and the operation specifying forward, reverse, or park.
[0034] The operating device 20 includes an operating mechanism for driving operations such as a steering wheel, a shift lever for shifting transmission gears, an accelerator pedal, a brake pedal, a turn signal lever, and push switches, as well as input devices such as a keyboard, a touch panel, and switches. The steering angle of the steering device is adjusted by the operation of the steering control unit by the occupant. At least one of the keyboard, touch panel, and switches receives various instructions and requests from the occupant, such as an automatic parking start request and an instruction to switch to remote parking mode. Details of these instructions and requests will be described later. The steering wheel is an example of a steering control unit. The steering wheel may be referred to as a handle. In this embodiment, automatic parking may be described as autonomous driving.
[0035] The shift lever is an example of a forward / reverse control unit. The forward / reverse control unit switches the vehicle 1 to at least forward, reverse, and park. The brake pedal is an example of a brake control unit. The brake control unit is a control unit for the brakes that control the speed of the vehicle 1. That is, the brake control unit receives instructions to decelerate the vehicle 1. The brake control unit also receives instructions to apply the brakes to the vehicle 1. The operating device 20 may constitute part of an HMI (Human Machine Interface) or IVI (In-Vehicle Infotainment).
[0036] The display device 22 is a display that outputs various images. The display device 22 is installed in a position visible to the occupants of the vehicle 1. Examples of displays include liquid crystal displays (LCDs), organic electro-luminescence (EL) displays, and projectors. The display device 22 and the operating device 20 may be integrated into a touch panel display by arranging them in a superimposed configuration. The display device 22 is an example of at least one of HMI and IVI.
[0037] Furthermore, the display device 22 is not limited to having only one display area. For example, the display device 22 may have multiple display areas.
[0038] Furthermore, the vehicle 1 may be equipped with multiple display devices 22. For example, the display device 22 may include a first display unit and a second display unit, the first and second display units being displays that output various images. The first and second display units are separate display devices 22. The first and second display units may be located in different positions within the vehicle 1. For example, the first display unit may function as an IVI (In-Vehicle Inspection Panel), and the second display unit may function as part of the instrument panel of the vehicle 1.
[0039] The wireless communication device 24 is a communication function that communicates wirelessly with an information processing device such as an external wireless communication terminal 3. The wireless communication device 24 may be provided in the vehicle control device 10.
[0040] Figure 2 is an explanatory diagram of an example arrangement of the sensor 14 and camera 16.
[0041] Vehicle 1 is equipped with, for example, four cameras 16 (cameras 16A to 16D) so as to be able to acquire the external conditions of vehicle 1 in at least four directions, such as the front, rear, right side, and left side of vehicle 1.
[0042] Specifically, for example, camera 16 includes camera 16A, camera 16B, camera 16C, and camera 16D. Camera 16A is positioned at the front of vehicle 1 and photographs the area in front of vehicle 1. Camera 16A may be referred to as the front camera. Camera 16B is positioned at the right side of vehicle 1 and photographs the area to the right of vehicle 1. Camera 16C is positioned at the left side of vehicle 1 and photographs the area to the left of vehicle 1. Camera 16D is positioned at the rear of vehicle 1 and photographs the area behind vehicle 1. Camera 16D may be referred to as the rear camera, or rear camera, etc.
[0043] The number of cameras 16 provided in the vehicle 1 is not limited to four. Further, for sensors included in the sensor 14 that detect objects such as lidars, radars, sonars, and ultrasonic sensors, it is preferable that the arrangement positions, the number of arrangements, and the like are adjusted in advance so as to be able to acquire the external conditions on each of the right side, left side, front, and rear of the vehicle 1. For example, as shown in FIG. 2, the sensor 14 includes sensors 14A to 14F. These sensors 14A to 14F are arranged in the vehicle 1 so as to be able to acquire the external conditions on each of the right side, left side, front, and rear of the vehicle 1.
[0044] Next, the configuration of the vehicle 1 will be described.
[0045] FIG. 3 is a schematic diagram showing an example of the external appearance configuration of the vehicle 1.
[0046] The vehicle 1 includes a vehicle body 2 and two pairs of wheels 23 arranged on the vehicle body 2 along a predetermined direction. The two pairs of wheels 23 include a pair of front tires 23F and a pair of rear tires 23R (see also FIG. 2). Note that FIG. 2 and FIG. 3 show, as an example, a case where the vehicle 1 is provided with four wheels 23. However, the number of wheels 23 provided in the vehicle 1 is not limited to this.
[0047] Next, the configuration in the vicinity of the driver's seat of the vehicle 1 according to the present embodiment will be described.
[0048] FIG. 4 is a diagram showing an example of the configuration in the vicinity of the driver's seat 24A of the vehicle 1 according to the present embodiment.
[0049] The vehicle 1 is provided with a driver's seat 24A and a front passenger seat 24B. In front of the driver's seat 24A, a windshield 25, a dashboard 26, a steering wheel 20A, operation buttons 20B, and a display device 22 are provided. Further, in the vicinity of the driver's seat 24A, a shift lever 20C which is a lever for transmission gear shifting is provided.
[0050] The steering wheel 20A, the operation buttons 20B, and the shift lever 20C are an example of the operation device 20.
[0051] The steering wheel 20A is located in front of the driver's seat 24A and can be operated by the occupant. The rotation angle of the steering wheel 20A, i.e., the steering angle, is electrically or mechanically linked to the change in direction of the front tires 23F, which are the steering wheels. The steering wheels may also be the rear tires 23R, or both the front tires 23F and the rear tires 23R may be steering wheels.
[0052] The operation button 20B is a button that can be operated by the passenger. The operation button 20B may also include a turn signal. The location of the operation button 20B is not limited to the example shown in Figure 4, and may be provided on the steering wheel 20A, for example. Although Figure 4 shows one operation button 20B, multiple operation buttons 20B may be provided. If the display device 22 also functions as a touch panel, the display device 22 may be an example of the operation device 20.
[0053] Returning to Figure 1, we continue the explanation.
[0054] The vehicle control device 10 is an electronic control unit that provides comprehensive control over all parts of the vehicle 1.
[0055] The vehicle control device 10 uses sensor information received from the sensor 14 and camera 16, respectively, as well as surrounding video footage, to control the movement control device 12 so that the driving state of the vehicle 1 is optimized. The vehicle control device 10 also controls the movement control device 12 to make the vehicle 1 drive autonomously.
[0056] The vehicle control device 10 includes a control unit 11. Part or all of the control unit 11 may be a software configuration realized through the cooperation of a processor and various programs stored in memory. Alternatively, part or all of the control unit 11 may be a hardware configuration realized by a dedicated circuit or the like.
[0057] The control unit 11 provides overall control over all parts of the vehicle 1.
[0058] In this embodiment, the control unit 11 performs information processing for assisting the parking of the vehicle 1.
[0059] The control unit 11 provides overall control over all parts of the vehicle 1.
[0060] In this embodiment, the control unit 11 is configured to switch or transition the driving mode to a teacher driving mode, an autonomous driving mode, or a remote parking mode based on input operations of the operating device 20 by the passenger. The driving modes that can be performed by the vehicle 1 may include various driving modes other than the teacher driving mode, autonomous driving mode, and remote parking mode.
[0061] The teacher driving mode is a mode for registering a teacher route when the vehicle 1 is driven autonomously. The teacher route is a route obtained by teacher driving from a predetermined position to a target position such as a target parking position. In teacher driving mode, the vehicle 1 is controlled by the operator's input. That is, in teacher driving mode, the control unit 11 receives an input from the operator 20, which is a driving operation by the operator, and controls the movement control device 12 to drive in accordance with that input. In other words, in teacher driving mode, the control unit 11 registers the teacher route based on surrounding images acquired by the camera 16 during the teacher driving of the vehicle 1 from a predetermined position to a target position, based on the operator's input.
[0062] The autonomous driving mode is a mode in which the vehicle 1 is driven autonomously. In this embodiment, the autonomous driving mode means a mode in which the vehicle 1 is driven autonomously to a target parking position based on a predetermined route. In the autonomous driving mode, the control unit 11 controls at least the steering to cause the movement control device 12 to control at least acceleration / deceleration and steering based on the predetermined route, thereby driving the vehicle 1 autonomously to the target parking position. The predetermined route may be a pre-registered teacher route or a route that can be driven to the target parking position, created by the control unit 11 during autonomous driving. In this embodiment, a configuration in which the predetermined route is a teacher route will be described as an example.
[0063] In autonomous driving mode, vehicle 1 is automatically controlled by the vehicle control device 10 without manual operation by the occupant. That is, in this embodiment, in autonomous driving mode, the control unit 11 controls the steering to make vehicle 1 autonomously drive from a predetermined position to a target parking position, based on the teacher route and surrounding images acquired by the camera 16 and sensor information. In this embodiment, the control by the movement control device 12 to make vehicle 1 autonomously drive to a target parking position by controlling acceleration, deceleration and steering based on a predetermined route such as the teacher route will be described as automatic parking control.
[0064] The remote parking mode is a mode in which the vehicle 1 is parked at a target parking position by remote control. Remote control means receiving remote control from an external source. In remote parking mode, the control unit 11 receives the operation received by the wireless communication terminal 3 as a remote control. Based on a predetermined route and the remote control from an external source, the control unit 11 causes the movement control device 12 to control at least acceleration / deceleration and steering to park the vehicle 1 at the target parking position. As described above, in this embodiment, a configuration in which the teacher route is used as the predetermined route will be described as an example.
[0065] Figure 5 is an explanatory diagram illustrating an example of the parking assistance system of this embodiment.
[0066] [In Teacher Driving Mode] First, the control of the control unit 11 in teacher driving mode will be explained in detail.
[0067] In teacher driving mode, the driver manually operates the control device 20 to perform a teacher drive from a predetermined position P1 to a target parking position P2. During the teacher drive, for example, the vehicle 1 travels in the direction D. The target parking position P2 is, for example, a parking lot, but is not limited to this. The predetermined position P1 can be any position of the driver in real space.
[0068] The driving path R traveled during the teacher driving is treated as the teacher path R1. As described above, the teacher path R1 is an example of a predetermined path. The path data 18A of the teacher path R1 is stored in the storage device 18. During the teacher driving, for example, the passenger operates the control device 20 to drive from a predetermined position P1 toward the target parking position P2. When the vehicle 1 travels from the predetermined position P1 toward the target parking position P2 during the teacher driving, the control unit 11 creates the path data 18A of the teacher path R1 along the driving direction D during the teacher driving of the driving path R.
[0069] In detail, when the control unit 11 receives a signal indicating an instruction to start the teacher driving mode, such as when the passenger operates the control device 20, it switches the driving mode to the teacher driving mode. Then, when in teacher driving mode, the control unit 11 performs the following processes.
[0070] The control unit 11 acquires sensor information from the sensor 14 indicating the driving state of the vehicle 1. The control unit 11 then estimates the current position of the vehicle 1 based on the temporal changes in the sensor values represented by the sensor information. For example, the control unit 11 calculates the amount of movement of the vehicle 1 from a reference position, such as the starting position when the training driving mode was started, based on the temporal changes in the vehicle speed and yaw rate represented by the sensor values, and estimates the current position of the vehicle 1 based on the amount of movement.
[0071] However, the accuracy of estimating the current position based on the amount of movement may be low. For this reason, the control unit 11 may use the result of correcting the estimated current position based on the video of the area around the vehicle 1 acquired by the camera 16 as the current position.
[0072] The control unit 11 sequentially stores the current position of the vehicle 1, which it has estimated in sequence as the vehicle 1 travels, into the storage device 18. Specifically, the control unit 11 defines the travel route R during the training mode, which is represented by the group of current positions sequentially estimated from the time it receives the instruction to start the training mode until it receives the instruction to end the training mode, as the training route R1, and stores the route data 18A representing the training route R1 in the storage device 18. The control unit 11 also stores map data 18B in the storage device 18.
[0073] Route data 18A consists of a group of location-specific driving information, which is the current position estimated sequentially during the teacher's run. The driving information includes an INDEX, driving position, bearing, driving direction, and reference driving information. The INDEX is the identification information for the driving information. The driving position is the estimated position of vehicle 1. The bearing indicates the orientation of vehicle 1 at that position. The driving direction indicates the direction of vehicle 1's movement at that position, and is represented, for example, by forward or reverse. The reference driving information is information that represents the driving state at that position. The reference driving information is, for example, information such as steering angle and vehicle speed detected at each position during the teacher's run.
[0074] Furthermore, during the training run of the vehicle 1, the control unit 11 creates map data 18B to estimate the current position of the vehicle 1 from the surrounding video footage captured by the camera 16. Methods such as SLAM (Simultaneous Localization and Mapping) are used to estimate the current position of the vehicle 1 from the surrounding video footage V.
[0075] Map data 18B is map data containing registered first feature points around vehicle 1 when it is traveling along the training route R1.
[0076] The first feature point is a feature point obtained by analyzing external conditions acquired by the sensor 14 during the teacher run, such as surrounding images captured by the camera 16 during the teacher run. In other words, in this embodiment, the feature point identified according to the external conditions acquired by the sensor 14 during the teacher run will be referred to as the first feature point.
[0077] For example, a feature point is a part of an object included in the real-world scene (e.g., a tree, wall, or pillar) from which a characteristic image pattern can be obtained by analyzing the video. This part is, for example, the edge portion of the object. As described above, in this embodiment, the feature points identified during the training run are referred to as first feature points. For this reason, the map data 18B contains multiple first feature points, and each first feature point is registered in a way that allows for individual identification by assigning an identification number.
[0078] Feature points are represented by feature point data that includes 3D location and feature quantities.
[0079] The three-dimensional position of a feature point is its three-dimensional position in real space, and can be represented, for example, in a three-dimensional Cartesian coordinate system (X, Y, Z).
[0080] The feature quantities of a feature point are characteristic quantities of that feature point that are represented by image analysis of the video. Examples of feature quantities of a first feature point include brightness and density in the video, SIFT (Scale Invariant Feature Transform) features, SURF (Speeded Up Robust Features) features, etc.
[0081] Map data 18B contains one first feature point registered for each identical three-dimensional location. Note that for each identical three-dimensional location, map data 18B may also contain multiple first feature points registered for each shooting position and shooting direction by the camera 16 at that location. Furthermore, the feature point data of the first feature point registered in map data 18B may further include image data of the object having that first feature point.
[0082] During the training run, the control unit 11 identifies the coordinates of the first feature point in the real-world scene using, for example, stereo photogrammetry based on the external conditions acquired by the sensor 14 during the training run. Specifically, the control unit 11 reads out multiple images taken at different timings that constitute the video included in the external conditions acquired by the sensor 14 during the training run, and associates the same first feature point that is commonly seen in these multiple images. Then, the control unit 11 estimates the provisional position of the vehicle 1 at the time these multiple images were taken, and identifies the provisional coordinates of the first feature point in the real-world scene using the principle of triangulation. Then, the control unit 11 performs bundle adjustment using, for example, the provisional position of the vehicle 1 and the provisional coordinates of the first feature point in the real-world scene as reference information, and calculates the official position of the vehicle 1 and the official coordinates of the first feature point in the real-world scene so as to minimize the reprojection error when projecting each first feature point in the real-world scene onto all the images. The control unit 11 then stores the registered map data 18B of the first feature point, which is represented by feature point data that includes the formal coordinates of the first feature point in the actual scene as a three-dimensional position, in the storage device 18.
[0083] The three-dimensional position of the first feature point registered in the map data 18B may be determined beforehand using LiDAR (Light Detection and Ranging) or a stereo camera, rather than by the SLAM method. However, it is preferable to use the SLAM method from the viewpoint of suppressing a decrease in position estimation accuracy.
[0084] As described above, the control unit 11 executes the above processing when in teacher driving mode. For this reason, when in teacher driving mode, the control unit 11 registers route data 18A and map data 18B in the storage device 18 based on the external conditions acquired by the sensor 14 during the teacher driving. In other words, the control unit 11 generates route data 18A of the teacher route R1 obtained by the teacher driving from a predetermined position P1 to the target parking position P2, and map data 18B in which the three-dimensional positions of each of the multiple first feature points around the vehicle 1 during driving along the teacher route R1 and the feature quantities of the first feature points are registered, based on the external conditions acquired by the sensor 14 during the teacher driving, and stores them in the storage device 18. For this reason, the storage device 18 stores route data 18A consisting of a group of driving information for each position, which is the current position estimated sequentially during the teacher driving, and map data 18B in which the first feature points observed for each position are registered.
[0085] [Autonomous Driving Mode] Next, the control of the control unit 11 in autonomous driving mode will be described in detail.
[0086] In autonomous driving mode, the control unit 11 controls acceleration, deceleration, and steering along a predetermined path such as the teacher path R1 obtained through teacher driving, and autonomously drives the vehicle 1 from a predetermined position P1 to a target parking position P2 based on the external conditions acquired by the sensor 14 during the teacher path R1 and autonomous driving. In autonomous driving mode, the control unit 11 performs steering control and forward / reverse acceleration / deceleration control of the vehicle 1, but at least a portion of the forward / reverse acceleration / deceleration control may be performed by the driver.
[0087] In detail, when the control unit 11 receives a signal indicating an automatic parking start request, such as through operation of the control device 20 by the passenger, it switches the driving mode to autonomous driving mode. Then, when the control unit 11 receives an automatic parking start request from the control device 20, it starts automatic parking control, which causes the movement control device 12 to control at least acceleration, deceleration and steering based on the teacher route R1 (a predetermined route) to autonomously drive the vehicle 1 to the target parking position P2. Upon initiation of automatic parking control by the movement control device 12, the vehicle 1 begins autonomous driving.
[0088] During autonomous driving, the control unit 11 controls at least acceleration, deceleration, and steering based on external conditions acquired by the sensor 14 to enable the vehicle 1 to drive autonomously.
[0089] In this embodiment, the control unit 11 reads route data 18A and map data 18B from the storage device 18 and controls the mobile control device 12 to perform autonomous driving along the teacher route R1 represented by the route data 18A.
[0090] The control unit 11 estimates the current position of the vehicle 1 based on the map data 18B and the surrounding video footage of the vehicle 1 acquired by at least one camera 16.
[0091] For example, the control unit 11 uses pattern matching, feature search, or the like to compare a second feature point identified from external conditions acquired by the sensor 14 during autonomous driving, including the surrounding image from the camera 16, with a first feature point identified from external conditions acquired by the sensor 14 during training driving, which is stored in the map data 18B.
[0092] The second feature point is a feature point obtained by analyzing external conditions acquired by the sensor 14 during autonomous driving, such as surrounding images captured by the camera 16 while autonomous driving along the teacher route R1. In other words, in this embodiment, the feature point identified according to the external conditions acquired by the sensor 14 during autonomous driving along the teacher route R1 will be described as the second feature point. The control unit 11 can identify the second feature point during autonomous driving along the teacher route R1 using the same method as the method used to identify the first feature point during teacher driving.
[0093] The control unit 11 randomly selects several (for example, 3 to 6) second feature points from among the second feature points identified during autonomous driving that can be matched with first feature points stored in the map data 18B.
[0094] The control unit 11 then estimates the current position of the vehicle 1 in real space based on the positions of these several second feature points in the video and the three-dimensional positions in real space of the first feature points registered in the map data 18B corresponding to these several second feature points. In this process, the control unit 11 estimates the current position of the vehicle 1 by solving a PnP problem using a known method such as Lambda Twist (for example, reference: Mikael Persson et al. "Lambda Twist: An Accurate Fast Robust Perspective Three Point (P3P) Solver.", ECCV 2018, pp 334-349, published in 2018, http: / / openaccess.thecvf.com / content_ECCV_2018 / papers / Mikael_Persson_Lambda_Twist_An_ECCV_2018_paper.pdf).
[0095] Through these processes, the control unit 11 estimates current position information representing the current position of the vehicle 1, based on the map data 18B and the surrounding video of the vehicle 1 acquired by at least one camera 16. This current position information includes the two-dimensional position (X coordinate, Y coordinate) of the vehicle 1 in real space, and information related to the orientation of the vehicle 1.
[0096] The control unit 11 then controls the movement control device 12 so that the estimated current position of the vehicle 1 is a position on the teacher route R1 represented by the route data 18A, thereby causing the vehicle 1 to autonomously travel along the teacher route R1 from a predetermined position P1 toward the target parking position P2. The control unit 11 then stops the vehicle 1 at the target parking position P2.
[0097] The control unit 11 provides feedback control to the movement control device 12 so that the vehicle 1 moves along the teacher route R1, based on the current position of the vehicle 1 estimated during autonomous driving along the teacher route R1 and each position on the teacher route R1 represented by the route data 18A.
[0098] Through these controls, in autonomous driving mode, the control unit 11 controls steering along a predetermined path, such as the teacher path R1 obtained through teacher driving, and makes the vehicle 1 drive autonomously based on the teacher path R1 and surrounding images acquired by the camera 16. In autonomous driving mode, the control unit 11 performs steering control and forward / reverse acceleration / deceleration control of the vehicle 1, but at least a part of the forward / reverse acceleration / deceleration control may be performed by the driver. Furthermore, as described above, in autonomous driving mode, the control unit 11 may control the movement control device 12 to drive along a predetermined path while creating a predetermined path to reach the target parking position P2 while avoiding obstacles around the vehicle 1 in known ways, without using the teacher path R1.
[0099] [Remote Parking Mode] Next, the control of the control unit 11 and the wireless communication terminal 3 in remote parking mode will be described in detail.
[0100] In remote parking mode, the control unit 11 accepts operations received by the wireless communication terminal 3 as remote operations. Based on the teacher path R1 and the remote operations from the wireless communication terminal 3 outside the vehicle 1, the control unit 11 causes the movement control device 12 to control at least acceleration, deceleration, and steering to drive the vehicle 1 to the target parking position P2 and park it.
[0101] In detail, in this embodiment, the control unit 11 attempts to establish a link with the remote operation application 30A of the wireless communication terminal 3.
[0102] Then, when the control unit 11 establishes a link with the remote operation application 30A and receives a signal indicating an instruction to switch to remote parking mode, such as through operation of the control device 20 by the passenger, it switches the driving mode to remote parking mode. In remote parking mode, the control unit 11, based on the user's remote operation to the wireless communication terminal 3 and the teacher path R1, causes the movement control device 12 to control at least acceleration, deceleration, and steering to park the vehicle 1 to the target parking position P2.
[0103] Figure 6 is a schematic diagram of an example of a screen 50 displayed on the wireless communication terminal 3.
[0104] Screen 50 is the screen that the remote operation application 30A of the wireless communication terminal 3 displays on the UI unit 36 of the wireless communication terminal 3 when it accepts remote operation from a user. Remote operation by a user using the wireless communication terminal 3 is performed from outside the vehicle 1 while the user carrying the wireless communication terminal 3 is outside the vehicle 1.
[0105] For example, screen 50 includes a remote operation area F. The remote operation area F is an operation area that accepts user input during remote operation. For example, a user operating the wireless communication terminal 3 will continue to move the point position TP, which is the user's touch position on screen 50, within the remote operation area F for the duration that the user desires to move vehicle 1. Also, if the user wants to stop vehicle 1, they will release the touch (contact) operation on the remote operation area F or stop the movement of the point position TP within the remote operation area F.
[0106] The remote operation application 30A transmits a "remote operation instruction received" signal to the movement control device 12 as long as the user is operating the point position TP within the remote operation area F and the movement operation of point position TP within the remote operation area F continues. The remote operation application 30A does not transmit the "remote operation instruction received" signal to the movement control device 12 when the user stops moving point position TP within the remote operation area F or when the touch operation to the remote operation area F is released. For this reason, the remote operation application 30A transmits a "remote operation instruction received" signal to the movement control device 12 as long as it is receiving a predetermined operation from the user to the UI unit 36.
[0107] Furthermore, the user's predetermined operation indicating that a remote operation instruction is being received is not limited to continuing to move the point position TP within the remote operation area F.
[0108] The vehicle's movement control device 12, while continuously receiving a remote operation instruction acceptance signal from the wireless communication terminal 3, controls steering along a predetermined path such as the teacher path R1, similar to the autonomous driving mode, and executes control to drive the vehicle 1 along the teacher path R1 and park it at the target parking position P2, based on the teacher path R1 and surrounding images acquired by the camera 16. On the other hand, in remote operation mode, the movement control device 12 stops the vehicle 1 from moving during periods when it does not receive a remote operation instruction acceptance signal from the wireless communication terminal 3.
[0109] Through these controls, in remote parking mode, the control unit 11 receives operations from the wireless communication terminal 3 as remote operations, and based on the teacher path R1 and the remote operations from the wireless communication terminal 3 from outside the vehicle 1, it causes the movement control device 12 to control at least acceleration / deceleration and steering to drive the vehicle 1 to the target parking position P2 and park it.
[0110] In this embodiment, the control unit 11 further assists in transitioning the vehicle 1, which is in autonomous driving mode, to remote operation mode by switching to autonomous driving mode. That is, in this embodiment, the control unit 11 transitions the vehicle 1 from autonomous driving mode to remote operation mode when predetermined conditions are met during autonomous driving.
[0111] The process by the control unit 11 to assist in transitioning from autonomous driving mode to remote operation mode will be explained in detail.
[0112] When the control unit 11 receives an automatic parking start request from the operating device 20, it instructs the movement control device 12 to start automatic parking control based on the teacher route R1 (a predetermined route).
[0113] More specifically, as described above, when the control unit 11 receives a signal indicating a request to start automatic parking, such as through operation of the control device 20 by the passenger, it switches the driving mode to autonomous driving mode. The control unit 11 then initiates automatic parking control by having the movement control device 12 control at least acceleration, deceleration, and steering, causing the vehicle 1 to autonomously drive along the teacher path R1 to the target parking position P2. Upon initiation of automatic parking control by the movement control device 12, the vehicle 1 begins autonomous driving along the teacher path R1.
[0114] Then, the control unit 11 causes the first information M1 to be displayed on the display device 22 while the vehicle 1 is autonomously driving.
[0115] Figure 7A is a schematic diagram of an example of screen 40A. Screen 40A is an example of screen 40 displayed on the display device 22.
[0116] When the control unit 11 switches to autonomous driving mode and autonomous driving of the vehicle 1 begins, it causes the display device 22 to display a screen 40A containing the first information M1. The screen 40A contains at least the first information M1.
[0117] The first piece of information M1 indicates that when the vehicle 1 stops due to the brake operation of the control device 20, and the control device 20 accepts the operation to specify parking, it will switch to remote parking mode.
[0118] Brake operation is, for example, a pressing operation on a brake operating part such as a brake pedal included in the operating device 20 to stop the vehicle 1.
[0119] The operation to designate parking is, for example, switching the shift lever included in the operating device 20 to the parking instruction position, which indicates parking.
[0120] Specifically, the first piece of information M1 indicates that when the vehicle 1 is stopped by the operation of the brake pedal or other brake control unit by the occupant, and the shift lever is operated to the parking position, the driving mode of the vehicle 1 switches to remote parking mode.
[0121] Figure 7A shows an example where the first information M1 is text indicating that the vehicle 1 stops when the brakes are applied by the control device 20 and the control device 20 accepts a parking instruction, thereby transitioning to remote parking mode. However, the first information M1 may also be a still image, an animated image, or an icon, and is not limited to text.
[0122] Furthermore, screen 40A may also include peripheral video V captured by camera 16. Figure 7A shows a configuration in which screen 40A includes peripheral video V1 and peripheral video V2 as peripheral video V. Peripheral video V1 is a top-view video of vehicle 1. Top view is an overhead view as if vehicle 1 were seen from above. Peripheral video V2 is a peripheral video V in the direction of travel of vehicle 1. The control unit 11 may also display a schematic image Ic of vehicle 1 at a position corresponding to the current position of vehicle 1 in the peripheral video V, with an inclination corresponding to the current posture of vehicle 1.
[0123] By causing the control unit 11 to display the first information M1 on the display device 22, the control unit 11 can inform the passenger that if the passenger performs the operation represented by the first information M1, the autonomously driving vehicle 1 can be switched to remote parking mode.
[0124] The control unit 11 may, while the vehicle 1 is autonomously driving, display the first information M1 on the display device 22 and attempt to establish a link with the remote operation application 30A of the wireless communication terminal 3. That is, the control unit 11 may, while the vehicle 1 is autonomously driving, display the first information M1 on the display device 22 and start a link establishment trial process to attempt to establish a link with the remote operation application 30A of the wireless communication terminal 3.
[0125] Furthermore, the control unit 11 may display the first information M1 or the second information M2 on the display device 22 when the vehicle 1 is autonomously driving and is within a predetermined distance from the target parking position P2, or within a predetermined time counting back from the expected arrival time at the target parking position P2.
[0126] This will be explained using Figure 5.
[0127] Being within a predetermined distance from the target parking position P2 means that the current position of vehicle 1 is within a predetermined distance when traced back from the target parking position P2 along the teacher path R1 toward the current position of vehicle 1 (direction of arrow D'). Alternatively, being within a predetermined distance from the target parking position P2 may also mean that the nearest position on the teacher path R1 to the current position of vehicle 1 is within a predetermined distance when traced back from the target parking position P2 along the teacher path R1 toward the current position of vehicle 1 (direction of arrow D').
[0128] The predetermined distance can be set in advance. Furthermore, the predetermined distance may be changed as appropriate by instructions from the passenger or other operator of the control device 20. The predetermined distance may, for example, be the length of the overlapping path B between the teacher path R1 and a rectangular area S2 of a predetermined size centered on the target parking position P2.
[0129] The estimated arrival time at the target parking position P2 is the expected arrival time when vehicle 1, which is autonomously driving along the teacher route R1, reaches the target parking position P2. The control unit 11 can calculate the estimated arrival time at the target parking position P2 using a known method.
[0130] "Within a predetermined time from the expected arrival time at the target parking position P2" means that the vehicle 1, which is autonomously driving along the teacher route R1, will reach the target parking position P2 within a predetermined time from the expected arrival time, which is a time in the future from the present, in the direction of going back in time (i.e., to the present).
[0131] The control unit 11 then causes the vehicle 1 to display the first information M1 or the second information M2 on the display device 22 when the vehicle 1 is autonomously driving and is within a predetermined distance from the target parking position P2, or within a predetermined time counting back from the expected arrival time at the target parking position P2.
[0132] Figure 7B is a schematic diagram of an example of screen 40B. Screen 40B is an example of screen 40 displayed on the display device 22.
[0133] The control unit 11 displays the first information M1 or the second information M2 on the display device 22 when the vehicle 1 is autonomously driving and is within a predetermined distance from the target parking position P2, or within a predetermined time counting back from the expected arrival time at the target parking position P2.
[0134] For example, in this case, the control unit 11 may display the screen 40A shown in Figure 7A on the display device 22, or it may display the screen 40B shown in Figure 7B on the display device 22.
[0135] Screen 40B includes at least the second information M2.
[0136] The second information M2, like the first information M1, indicates that the vehicle 1 will switch to remote parking mode when the brake operation of the operating device 20 is performed and the operating device 20 accepts the operation to specify parking. The second information M2 further includes information indicating that the vehicle is within a predetermined distance from the target parking position P2, or within a predetermined time counting back from the expected arrival time at the target parking position P2.
[0137] Figure 7B shows an example where the second information M2 is in the form of text. However, the second information M2 may also be a still image, an animated image, or an icon, and is not limited to text.
[0138] Furthermore, screen 40B may also include peripheral video V captured by camera 16. Figure 7B shows a configuration in which screen 40B includes peripheral video V1 and peripheral video V2 as peripheral video V. The control unit 11 may also display a schematic image Ic of vehicle 1 at a position corresponding to the vehicle 1's current position in the peripheral video V, with an inclination corresponding to the vehicle 1's current posture.
[0139] By causing the control unit 11 to display the first information M1 on the display device 22, the control unit 11 can inform the passenger that if the passenger performs the operation represented by the second information M2, the autonomously driving vehicle 1 can be switched to remote parking mode.
[0140] Then, when the control unit 11 receives an operation represented by the first information M1 or the second information M2 while the vehicle 1 is autonomously driving, it switches the driving mode of the vehicle 1 to remote parking mode.
[0141] In other words, when the vehicle 1 is driving autonomously, and the vehicle 1 stops due to a brake operation by the occupant using the operating device 20 (brake operation unit), and the operating device 20 (shift lever) receives a parking instruction, the control unit 11 switches the vehicle 1 to remote parking mode.
[0142] In detail, when the vehicle 1 is autonomously driving and the vehicle 1 stops due to a brake operation by the passenger using the control device 20, the control unit 11 determines whether a link has been established with the remote control application 30A of the wireless communication terminal 3. Alternatively, when the vehicle 1 is autonomously driving and the vehicle 1 stops due to a brake operation by the passenger using the control device 20, and the control device 20 receives a parking instruction, the control unit 11 may also determine whether a link has been established with the remote control application 30A of the wireless communication terminal 3.
[0143] The control unit 11 then displays the third information M3 on the display device 22 if a link with the remote operation application 30A has been established. On the other hand, the control unit 11 displays the fourth information M4 on the display device 22 if a link with the remote operation application 30A has not been established.
[0144] The third piece of information, M3, indicates that a link has been established with the remote control application 30A. The fourth piece of information, M4, prompts the wireless communication terminal 3 to establish a link with the remote control application 30A.
[0145] Figure 7C is a schematic diagram of an example of screen 40C. Screen 40C is an example of screen 40 displayed on the display device 22.
[0146] When the vehicle 1 is autonomously driving, if the vehicle 1 stops due to a brake operation by the passenger using the control device 20, and the control device 20 receives a parking instruction, the control unit 11 determines whether a link has been established with the remote operation application 30A of the wireless communication terminal 3. If it determines that a link has been established, it displays a screen 40C containing third information M3 on the display device 22. The screen 40C contains at least the third information M3.
[0147] Figure 7C shows an example where the third piece of information M3 is text. However, the third piece of information M3 may also be a still image, an animated image, or an icon, and is not limited to text.
[0148] Furthermore, screen 40C may also include peripheral video V captured by camera 16. Figure 7C shows a configuration in which screen 40C includes peripheral video V1 and peripheral video V2 as peripheral video V. The control unit 11 may also display a schematic image Ic of vehicle 1 at a position corresponding to the vehicle 1's current position in the peripheral video V, with an inclination corresponding to the vehicle 1's current posture.
[0149] By causing the control unit 11 to display the third information M3 on the display device 22, the control unit 11 can inform the passenger that a link with the remote control application 30A has been established.
[0150] If a link with the remote control application 30A has been established, the control unit 11 can switch the vehicle 1 to remote parking mode.
[0151] Figure 7D is a schematic diagram of an example of screen 40D. Screen 40D is an example of screen 40 displayed on the display device 22.
[0152] When the vehicle 1 is autonomously driving, if the vehicle 1 stops due to a brake operation by the passenger using the control device 20, and the control device 20 receives a parking instruction, the control unit 11 determines whether a link has been established with the remote operation application 30A of the wireless communication terminal 3. If it determines that a link has not been established, it displays a screen 40D containing the fourth information M4 on the display device 22. The screen 40D contains at least the fourth information M4.
[0153] Figure 7D shows an example where the fourth piece of information M4 is text. However, the fourth piece of information M4 may also be a still image, an animated image, or an icon, and is not limited to text.
[0154] Furthermore, screen 40D may also include peripheral video V captured by camera 16. Figure 7D shows a configuration in which screen 40D includes peripheral video V1 and peripheral video V2 as peripheral video V. The control unit 11 may also display an image Ic, which is a schematic representation of vehicle 1, at a position corresponding to the vehicle 1's current position in the peripheral video V, with an inclination corresponding to the vehicle 1's current posture.
[0155] By displaying the fourth information M4 on the display device 22, the control unit 11 can prompt the passenger to perform an operation to establish a link with the remote operation application 30A. Reasons why the link between the vehicle control device 10 and the remote operation application 30A cannot be established include the wireless communication terminal 3 not being powered on, or the remote operation application 30A not being installed on the wireless communication terminal 3. For this reason, it is preferable that the fourth information M4 includes information that prompts the passenger to resolve these issues. By displaying the fourth information M4 on the display device 22, the control unit 11 can prompt the passenger to resolve these issues that prevent the link with the remote operation application 30A from being established.
[0156] The control unit 11 then switches the vehicle 1 to remote parking mode when a link is established between the movement control device 12 and the remote operation application 30A of the wireless communication terminal 3.
[0157] In detail, when the vehicle 1 is autonomously driving, and the vehicle 1 stops due to a brake operation by the passenger using the control device 20, and the control device 20 receives a parking instruction, the control unit 11 determines whether a link has been established with the remote operation application 30A of the wireless communication terminal 3. If it determines that a link has been established, it displays the remote transition button B (see Figure 7C) on the display device 22.
[0158] Remote transition button B is a button image used by the passenger to instruct the vehicle to switch to remote parking mode. The passenger inputs the instruction to switch to remote parking mode by operating the button image displayed on the display device 22. When the control unit 11 determines that remote transition button B has been operated, it should switch vehicle 1 to remote parking mode.
[0159] In other words, the control unit 11 may switch to remote parking mode if, while the vehicle 1 is autonomously driving, the vehicle 1 stops due to a brake operation by the passenger using the control device 20, the control device 20 receives an operation to specify parking, and the control device 20 receives an instruction from the passenger to switch to remote parking mode.
[0160] When vehicle 1 switches to remote parking mode, the control unit 11, based on the teacher route R1 and remote operation from an external source, causes the movement control device 12 to control at least acceleration, deceleration, and steering to drive vehicle 1 to the target parking position P2 and park it.
[0161] In detail, as described above, the control unit 11 accepts operations received by the wireless communication terminal 3 as remote operations. Specifically, for example, a passenger in vehicle 1 gets out of vehicle 1 and moves outside of vehicle 1. The user, who is the passenger who got out of vehicle 1, then performs a movement operation to move the touched point position TP on the screen 50 of the wireless communication terminal 3 within the remote operation area F. The remote operation application 30A of the wireless communication terminal 3 continues to accept the movement operation of point position TP within the remote operation area F and transmits a remote operation instruction received signal to the movement control device 12, indicating that a remote operation instruction is being received. Furthermore, if the user stops moving point position TP within the remote operation area F, or if the touch operation to the remote operation area F is canceled, the remote operation application 30A stops transmitting the remote operation instruction received signal to the movement control device 12.
[0162] While the vehicle 1's movement control device 12 continues to receive a remote operation instruction acceptance signal from the wireless communication terminal 3's remote operation application 30A, it controls steering along a predetermined path such as the teacher path R1, similar to the autonomous driving mode, and controls the vehicle 1 to move and park at the target parking position P2 based on the teacher path R1 and surrounding images acquired by the camera 16. On the other hand, in remote operation mode, the movement control device 12 stops the vehicle 1 from moving during periods when it does not receive a remote operation instruction acceptance signal from the wireless communication terminal 3.
[0163] Through these controls, in remote parking mode, the control unit 11 accepts operations received by the wireless communication terminal 3 as remote operations, and based on the teacher path R1 and the remote operations from the wireless communication terminal 3 from outside the vehicle 1, it causes the movement control device 12 to control at least acceleration / deceleration and steering to park the vehicle 1 to the target parking position P2.
[0164] On the other hand, if the vehicle 1 does not stop due to braking by the control device 20 during autonomous driving, the control unit 11 can instruct the motion control device 12 to continue the autonomous driving mode. In other words, in this case, the control unit 11 can instruct the motion control device 12 to control at least acceleration, deceleration, and steering based on the teacher path R1 to park the vehicle 1 autonomously to the target parking position P2.
[0165] Next, an example of the information processing flow performed by the movement control device 12 of this embodiment will be described.
[0166] Figure 8 is a flowchart showing an example of the information processing flow performed by the vehicle control device 10 of this embodiment during autonomous driving.
[0167] The control unit 11 determines whether the operating device 20 has received an automatic parking start request (step S100). The control unit 11 makes the determination in step S100 by determining whether it has received a signal indicating an automatic parking start request, for example, by operating the operating device 20 by a passenger. The signal indicating an automatic parking start request is output from the operating device 20 (display device 22) to the control unit 11 by, for example, an instruction from a passenger to operate an operation button for receiving an automatic parking start request included in the operating device 20, or an instruction to operate the said operation button displayed on the display device 22.
[0168] The control unit 11 repeatedly makes a negative judgment (step S100: No) until it makes an affirmative judgment (step S100: Yes) in step S100. If the control unit 11 makes an affirmative judgment (step S100: Yes) in step S100, it proceeds to step S102.
[0169] In step S102, the control unit 11 switches the driving mode to autonomous driving mode and causes the vehicle 1 to start autonomous driving (step S102). Specifically, the control unit 11 causes the movement control device 12 to control at least acceleration, deceleration and steering to start automatic parking control, causing the vehicle 1 to autonomously drive along the teacher path R1 to the target parking position P2. Upon initiation of automatic parking control by the movement control device 12, the vehicle 1 starts autonomous driving along the teacher path R1.
[0170] Next, the control unit 11 determines whether the current driving mode of the vehicle 1 is remote operation mode (step S104). If the determination in step S104 is positive (step S104: Yes), the process proceeds to step S134, which will be described later. If the determination in step S104 is negative (step S104: No), the process proceeds to step S106.
[0171] In step S106, the control unit 11 determines whether or not the remote transition button B is currently displayed on the display device 22 (step S106). If the determination in step S106 is positive (step S106: Yes), the process proceeds to step S126, which will be described later. If the determination in step S106 is negative (step S106: No), the process proceeds to step S108.
[0172] In step S108, the first information M1 is displayed on the display device 22 (step S108). As a result of the process in step S108, for example, a screen 40A containing the first information M1 is displayed on the display device 22 (see Figure 7A).
[0173] Furthermore, the control unit 11 performs a link establishment trial process to attempt to establish a link with the remote operation application 30A of the wireless communication terminal 3 (step S110).
[0174] Next, the control unit 11 determines whether the current position of the vehicle 1 is within a predetermined distance from the target parking position P2, or, if so, whether the current time is within a predetermined time counting back from the assumed arrival time at the target parking position P2 (step S112). If the determination in step S112 is negative (step S112: No), the process proceeds to step S130, which will be described later. If the determination in step S112 is positive (step S112: Yes), the process proceeds to step S114.
[0175] In step S114, the control unit 11 causes the second information M2 to be displayed on the display device 22 (step S114). As a result of the processing in step S114, for example, the display device 22 displays a screen 40B including the second information M2 (see Figure 7B).
[0176] Next, the control unit 11 determines whether the vehicle 1 has stopped due to the brake operation of the operating device 20 (brake operation unit) by the passenger, and whether the operating device 20 (shift lever) has accepted the operation to specify parking (step S116). If the determination in step S116 is negative, the process proceeds to step S136, which will be described later. If the determination in step S116 is positive (step S116: Yes), the process proceeds to step S118.
[0177] In step S118, the control unit 11 determines whether or not a link has been established with the remote operation application 30A of the wireless communication terminal 3 (step S118). The determination in step S118 can be performed by a known communication establishment determination process.
[0178] If a negative result is determined in step S118 (step S118: No), the process proceeds to step S120. In step S120, the control unit 11 displays the fourth information M4 on the display device 22 (step S120). As a result of the processing in step S120, for example, a screen 40D containing the fourth information M4 is displayed on the display device 22 (see Figure 7D). Then, the process returns to step S118.
[0179] If a positive determination is made in step S118 (step S118: Yes), the process proceeds to step S122. In step S122, the control unit 11 displays the third information M3 on the display device 22 (step S122). As a result of the processing in step S122, for example, a screen 40C containing the third information M3 is displayed on the display device 22 (see Figure 7C). Then, the process proceeds to step S124.
[0180] In step S124, the control unit 11 displays the remote transition button B on the display device 22 (step S124).
[0181] Next, the control unit 11 determines whether the remote transition button B displayed on the display device 22 in step S124 was pressed by the passenger (step S126). The control unit 11 makes the determination in step S126 by determining whether the display area of the remote transition button B displayed on the display device 22 has been operated. If the determination in step S126 is negative (step S126: No), the system proceeds to step S130, which will be described later. If the determination in step S126 is positive (step S126: Yes), the system proceeds to step S128.
[0182] In step S128, the control unit 11 cancels the display of the remote control transition button and switches the driving mode of the vehicle 1 to remote parking mode (step S128).
[0183] Next, in step S130, the control unit 11 performs driving control along the path (step S130). Specifically, the control unit 11 performs automatic parking control by having the movement control device 12 control at least acceleration, deceleration, and steering so that the vehicle 1 autonomously drives along the teacher path R1 to the target parking position P2. Upon commencement of automatic parking control by the movement control device 12, the vehicle 1 autonomously drives along the teacher path R1.
[0184] Next, in step S132, the control unit 11 determines whether or not the vehicle 1 has reached the target parking position P2 (step S132). For example, the control unit 11 makes the determination in step S132 by determining whether or not the current position of the vehicle 1 coincides with the target parking position P2. If the determination in step S132 is negative (step S132: No), the process returns to step S104. If the determination in step S132 is positive (step S132: Yes), the routine terminates.
[0185] On the other hand, if a positive decision is made in step S104 (step S104: Yes), the process proceeds to step S134. In step S134, the control unit 11 accepts the operation received by the wireless communication terminal 3 as a remote operation. Specifically, the control unit 11 determines whether or not it is currently receiving a remote operation instruction from the remote operation application 30A of the wireless communication terminal 3 (step S134). The control unit 11 makes the decision in step S134 by determining whether or not it is currently receiving a remote operation instruction acceptance signal from the remote operation application 30A.
[0186] If a positive determination is made in step S134 (step S134: Yes), the control unit 11 controls the steering along the teacher path R1 and executes control to park the vehicle 1 at the target parking position P2 based on the teacher path R1 and the surrounding image V acquired by the camera 16 (step S136). Then, the process proceeds to step S132.
[0187] On the other hand, if a negative determination is made in step S134 (step S134: No), the control unit 11 stops the vehicle 1 from moving (step S138). Then, the process proceeds to step S132.
[0188] As described above, the vehicle control device 10 of this embodiment is mounted on a vehicle 1 equipped with an operating device 20, a camera 16, a display device 22 visible to the occupant, and a movement control device 12. The operating device 20 accepts at least brake operation and operation to specify forward, reverse, or park. The camera 16 acquires a video image V of the surroundings of the vehicle 1. The movement control device 12 controls at least acceleration / deceleration and steering.
[0189] When the operating device 20 receives an automatic parking start request, the vehicle control device 10 initiates automatic parking control, which causes the movement control device 12 to control at least acceleration, deceleration, and steering based on a predetermined route (teacher route R1) to autonomously drive the vehicle 1 to the target parking position P2.
[0190] Furthermore, the vehicle control device 10 displays the first information M1 on the display device 22 while the vehicle 1 is autonomously driving. The first information M1 indicates that the vehicle 1 will stop when the brakes are applied by the operating device 20, and that the operating device 20 will switch to remote parking mode when it receives a parking instruction.
[0191] Furthermore, when the vehicle 1 is autonomously driving, if the vehicle 1 stops due to a brake operation by the operating device 20, and the operating device 20 receives a parking instruction, the vehicle control device 10 switches to remote parking mode.
[0192] Furthermore, if the vehicle 1 does not stop due to braking by the operating device 20 while the vehicle 1 is autonomously driving, the vehicle control device 10 causes the movement control device 12 to control at least acceleration, deceleration, and steering based on a predetermined route (teacher route R1) to park the vehicle 1 autonomously to the target parking position P2.
[0193] Thus, the vehicle control device 10 of this embodiment displays first information M1 on the display device 22 indicating that when the vehicle 1 is autonomously driving, the vehicle 1 stops due to a brake operation by the operating device 20, and the operating device 20 accepts a parking instruction, it will switch to remote parking mode.
[0194] Therefore, the vehicle control device 10 of this embodiment can provide the occupant with the conditions and operation details for transitioning the vehicle 1 to remote parking mode while the vehicle 1 is autonomously driving. Furthermore, when the vehicle 1 stops and the operation detail of specifying parking is received, the vehicle control device 10 of this embodiment can transition the vehicle 1 to remote parking mode.
[0195] Therefore, the vehicle control device 10 of this embodiment can provide a method for transitioning from autonomous driving to remote parking mode while the vehicle 1 is autonomously driving, and can support a suitable transition to remote parking mode during autonomous driving.
[0196] Therefore, the vehicle control device 10 of this embodiment can provide suitable parking assistance.
[0197] Furthermore, the vehicle control device 10 of this embodiment is particularly useful for assisting the parking of a vehicle 1 to a target parking position P2 in a particularly narrow parking space, such as a mechanical parking garage.
[0198] Furthermore, the vehicle control method in this embodiment may be any vehicle control method executed by the vehicle control device 10.
[0199] Next, the hardware configuration of the vehicle control device 10 and the control unit 30 of the wireless communication terminal 3 in this embodiment will be described.
[0200] Figure 9 is a block diagram showing examples of the hardware configurations of the vehicle control device 10 and the control unit 30.
[0201] The vehicle control device 10 and the control unit 30 have a CPU (Central Processing Unit) 11A, ROM (Read Only Memory) 11B, RAM (Random Access Memory) 11C, and I / F 11D for connecting to various devices, etc., all interconnected by a bus 11E, resulting in a hardware configuration that utilizes a standard computer.
[0202] The CPU 11A is an arithmetic unit that controls the overall processing of the vehicle control device 10 and the control unit 30. The RAM 11C stores data necessary for various processes performed by the CPU 11A. The ROM 11B stores programs and the like that implement the various processes performed by the CPU 11A. The I / F 11D is an interface that connects to external devices and external terminals via communication lines, etc., and is used to send and receive data between the connected external devices and external terminals.
[0203] The programs for executing the various processes described above, which are performed by the vehicle control device 10 and the control unit 30, are provided pre-installed in a ROM 11B or the like. The programs for executing the vehicle control method performed in this embodiment may also be provided as files in an installable or executable format for these devices, recorded on a computer-readable recording medium such as a CD-ROM, flexible disk (FD), CD-R, or DVD (Digital Versatile Disc).
[0204] Furthermore, the program for executing the vehicle control method performed in this embodiment may be stored on a computer connected to a network such as the Internet and provided by allowing download via the network. Alternatively, the program for executing the vehicle control method performed in this embodiment may be provided or distributed via a network such as the Internet.
[0205] While embodiments of this disclosure have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be carried out in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments are included in the scope and spirit of the invention, as well as in the claims and their equivalents.
[0206] 1 Vehicle 3 Wireless communication terminal 10 Vehicle control device 11 Control unit 12 Movement control device 14 Sensor 16 Camera 20 Operating device 22 Display device 24 Wireless communication device 30A Remote control application
Claims
1. A vehicle control method to be performed by a vehicle control device mounted on a vehicle, comprising: an operating device that receives at least a brake operation and an operation to specify forward, reverse, or park from the occupant; a camera that acquires surrounding images; a display device that is visible to the occupant; and a movement control device that controls at least acceleration, deceleration, and steering, wherein when the operating device receives an automatic parking start request, automatic parking control is started, causing the movement control device to control at least acceleration, deceleration, and steering based on a predetermined route, to autonomously drive the vehicle to a target parking position; the display device is shown that the vehicle will switch to remote parking mode when the vehicle stops due to the brake operation of the operating device and the operating device receives an operation to specify parking during the autonomous driving of the vehicle; and the remote parking mode is switched when the vehicle stops due to the brake operation of the operating device and the operating device receives an operation to specify parking during the autonomous driving of the vehicle. A vehicle control method comprising: if, during the autonomous driving of the vehicle, the vehicle does not stop due to at least the brake operation of the operating device, the movement control device controls at least acceleration, deceleration and steering based on the predetermined path to park the vehicle autonomously to the target parking position.
2. A vehicle control method according to claim 1, wherein, during autonomous driving of the vehicle, the vehicle stops due to the brake operation of the operating device, the operating device receives an operation to specify parking, and the operating device receives an instruction from the passenger to switch to remote parking mode, the vehicle switches to the remote parking mode.
3. A vehicle control method according to claim 1, wherein, when the vehicle is autonomously driving and within a predetermined distance from the target parking position, the vehicle stops due to the brake operation of the operating device, and when the operating device receives an operation to specify parking, the display device displays that the vehicle will transition to the remote parking mode.
4. A vehicle control method according to claim 1, wherein, while the vehicle is autonomously driving, and within a predetermined time period counting back from the expected arrival time at the target parking position, the vehicle stops due to the brake operation of the operating device, and when the operating device receives an operation to specify parking, the display device displays that the vehicle will transition to the remote parking mode.
5. A vehicle control method according to claim 1, wherein, during autonomous driving of the vehicle, if the vehicle stops due to the brake operation of the operating device and the operating device receives an operation to specify parking, the vehicle is switched to the remote parking mode, and after switching to the remote parking mode, the vehicle is parked at the target parking position by having the movement control device control at least acceleration / deceleration and steering based on the predetermined route and remote operation from an external source.
6. A vehicle control method according to claim 5, wherein the vehicle control device comprises a wireless communication device that communicates wirelessly with an external wireless communication terminal, and the operation received by the wireless communication terminal is treated as the remote operation, causing the vehicle control device to control at least acceleration / deceleration and steering to park the vehicle to the target parking position.
7. A vehicle control method according to claim 6, wherein the wireless communication terminal includes a remote operation application for remote operation, and when the vehicle stops due to the brake operation of the operating device while the vehicle is autonomously driving, and a link has been established between the vehicle control device and the remote operation application of the wireless communication terminal, the display device is made to display that a link has been established with the remote operation application.
8. A vehicle control method according to claim 7, wherein, while the vehicle is autonomously driving, the display device displays that the vehicle will switch to the remote parking mode when the vehicle stops due to the brake operation of the operating device and the operating device accepts the operation to specify parking, and the method attempts to establish a link with the remote operation application of the wireless communication terminal.
9. A vehicle control method according to claim 6, wherein the wireless communication terminal is equipped with a remote operation application for remote operation, and when the vehicle stops due to the brake operation of the operating device while the vehicle is autonomously driving, and a link between the vehicle control device and the remote operation application of the wireless communication terminal has not been established, the display device is configured to display information prompting the establishment of a link between the wireless communication terminal and the remote operation application.
10. A vehicle control method according to claim 9, wherein, during autonomous driving of the vehicle, when the vehicle stops due to the brake operation of the operating device and the operating device accepts the operation to designate parking, and a link has not been established between the vehicle control device and the remote operation application of the wireless communication terminal, information prompting the establishment of a link with the remote operation application of the wireless communication terminal is displayed on the display device, and thereafter, when a link is established between the vehicle control device and the remote operation application of the wireless communication terminal, the vehicle is switched to the remote parking mode.
11. A vehicle control device mounted on a vehicle comprising: an operating device that receives at least a brake operation and an operation to specify forward, reverse, or park from the occupant; a camera that acquires surrounding images; a display device visible to the occupant; and a movement control device that controls at least acceleration, deceleration, and steering, wherein when the operating device receives an automatic parking start request, it initiates automatic parking control to autonomously drive the vehicle to a target parking position based on a predetermined route, by having the movement control device control at least acceleration, deceleration, and steering; and during the autonomous driving of the vehicle, it displays on the display device that the vehicle will switch to remote parking mode when the vehicle stops due to the brake operation of the operating device and the operating device receives an operation to specify parking, and during the autonomous driving of the vehicle, it switches to remote parking mode when the vehicle stops due to the brake operation of the operating device and the operating device receives an operation to specify parking. A vehicle control device that, during autonomous driving of the vehicle, if the vehicle does not stop due to at least the brake operation of the operating device, causes the movement control device to control at least acceleration / deceleration and steering based on the predetermined path, causing the vehicle to park autonomously to the target parking position.
12. A vehicle control device according to claim 11, wherein, during autonomous driving of the vehicle, when the vehicle stops due to the brake operation of the operating device, the operating device receives an operation to specify parking, and the operating device receives an instruction from the passenger to switch to remote parking mode, the vehicle control device switches to the remote parking mode.
13. A vehicle control device according to claim 11, wherein, when the vehicle is autonomously driving and within a predetermined distance from the target parking position, the vehicle stops due to the brake operation of the operating device, and the operating device accepts the operation to specify parking, the vehicle will transition to the remote parking mode, and this information is displayed on the display device.
14. A vehicle control device according to claim 11, wherein, while the vehicle is autonomously driving, and within a predetermined time period counting back from the expected arrival time at the target parking position, the vehicle stops due to the brake operation of the operating device, and the operating device accepts the operation to specify parking, the vehicle will transition to the remote parking mode, and this information is displayed on the display device.
15. A vehicle control device according to claim 11, wherein, during autonomous driving of the vehicle, if the vehicle stops due to the brake operation of the operating device and the operating device receives an operation to specify parking, the vehicle is switched to the remote parking mode, and after switching to the remote parking mode, the vehicle is parked at the target parking position by having the movement control device control at least acceleration / deceleration and steering based on the predetermined route and remote operation from an external source.
16. A vehicle control device according to claim 15, wherein the vehicle control device comprises a wireless communication device that communicates wirelessly with an external wireless communication terminal, and the operation received by the wireless communication terminal is treated as the remote operation, causing the vehicle control device to control at least acceleration / deceleration and steering to park the vehicle to the target parking position.
17. A vehicle control device according to claim 16, wherein the wireless communication terminal includes a remote operation application for remote operation, and when the vehicle stops due to the brake operation of the operating device while the vehicle is autonomously driving, and a link has been established between the vehicle control device and the remote operation application of the wireless communication terminal, the vehicle control device displays on the display device that a link has been established with the remote operation application.
18. A vehicle control device according to claim 17, wherein, during autonomous driving of the vehicle, the vehicle stops due to the brake operation of the operating device, and when the operating device receives an operation to specify parking, the vehicle transitions to the remote parking mode, and the vehicle control device attempts to establish a link with the remote operation application of the wireless communication terminal.
19. A vehicle control device according to claim 16, wherein the wireless communication terminal includes a remote operation application for remote operation, and when the vehicle stops due to the brake operation of the operating device while the vehicle is autonomously driving, and a link between the vehicle control device and the remote operation application of the wireless communication terminal has not been established, the vehicle control device displays information on the display device prompting the establishment of a link between the wireless communication terminal and the remote operation application.
20. A vehicle control device according to claim 19, wherein, during autonomous driving of the vehicle, when the vehicle stops due to the brake operation of the operating device and the operating device accepts the operation to designate parking, and a link has not been established between the vehicle control device and the remote operation application of the wireless communication terminal, the vehicle control device displays information on the display device prompting the establishment of a link with the remote operation application of the wireless communication terminal, and thereafter, when a link has been established between the vehicle control device and the remote operation application of the wireless communication terminal, the vehicle control device transitions to the remote parking mode.
