Boarding assistance system

Abstract

Provided is a boarding assistance system capable of performing vehicle control for assisting boarding without delay. Specifically, the position of a portable device 16 corresponding to the key of a vehicle 2 is detected, an image 50 captured by a camera provided in the vehicle 2 is acquired, a passenger of the vehicle 2 included in the image 50 is identified using the detection result of the position of the portable device 16, the intention to board of the passenger is determined based on the behaviour of the identified passenger, and vehicle control for assisting boarding of the vehicle is performed when it is determined that the passenger has the intention to board.

Description

Boarding assistance system

[0001] The present invention relates to a boarding assistance system that assists boarding a vehicle.

[0002] Conventionally, various systems for assisting occupants to board a vehicle have been proposed for occupants who board the vehicle. In particular, as assistance for automatically unlocking and opening the vehicle door when a person gets on the vehicle on the system side, for example, Japanese Patent Laid-Open No. 2022-134315 discloses that an on-board camera of the vehicle captures an image of the occupant of the vehicle, and detects the occupant's behavior (position, movement trajectory, orientation, etc.) to determine the occupant's intention to board. When it is determined that the occupant intends to board, specifically when the occupant performs a predetermined leg movement to position themselves parallel to the vehicle, a technology for unlocking the door has been proposed.

[0003] Japanese Patent Laid-Open No. 2022-134315

[0004] Here, in order to capture an image of the vehicle occupant located outside the vehicle with an on-board camera and confirm the occupant's intention to board as in the above Patent Document 1, it is desirable to install the imaging device so that it can capture an image as wide a range as possible, so that the occupant can be detected no matter what direction the occupant approaches the vehicle from.

[0005] However, on the other hand, if the imaging range is widened, the number of objects to be subjected to image recognition processing on the captured image increases, so the processing time becomes longer and the processing load also increases. For example, if the processing time becomes long, there is a problem that the determination of the occupant's intention to board is delayed, and the unlocking of the door is also delayed. In addition, widening the imaging range increases the possibility that people other than the occupant are included in the captured image, which also causes a problem that the accuracy of determining the occupant's intention to boarding decreases.

[0006] The present invention has been made to solve the above-mentioned conventional problems. By using the detection result of the key position in combination with the determination of the occupant's intention to board by an imaging device, it is possible to accurately grasp the occupant's intention to board at the earliest possible timing, and an object of the present invention is to provide a boarding assistance system that enables vehicle control for assisting boarding without delay.

[0007] To achieve the above objective, the passenger boarding assistance system according to the present invention includes: a key position detection unit for detecting the position of the vehicle key; an image acquisition unit for acquiring an image captured by an imaging device provided by the vehicle; an occupant identification unit for identifying the occupants of the vehicle included in the image captured using the detection result of the key position detection unit; an occupant boarding intention determination unit for determining the occupant's intention to board based on the behavior of the occupant identified by the occupant identification unit; and a vehicle control unit for performing vehicle control to assist boarding the vehicle when it is determined that the occupant intends to board. The "vehicle control to assist boarding the vehicle" may be a control to unlock the doors, a control to open the doors, or both, or a control to turn on the engine or turn on the interior lights. Furthermore, the "vehicle key" is a device equipped with the function of locking or unlocking the vehicle doors, and is not limited to a key that is physically inserted into the door to lock or unlock it, but also includes an electronic key equipped with the function of locking or unlocking by communication with the vehicle, and an electronic device (e.g., a smartphone) on which an application that performs these functions is installed.

[0008] According to the boarding assistance system of the present invention having the above configuration, by combining the detection result of the key position with the determination of the occupant's intention to board the vehicle by the imaging device, it becomes possible to accurately grasp the occupant's intention to board the vehicle as early as possible. As a result, it becomes possible to perform vehicle control to assist boarding without delay.

[0009] This is a schematic diagram of the vehicle according to this embodiment. This is a block diagram showing the configuration of the passenger assistance system according to this embodiment. This is a flowchart of the passenger assistance processing program according to this embodiment. This is a diagram illustrating typical movements of an occupant. This is a diagram illustrating the narrowing of the image processing area. This is a diagram illustrating skeletal detection by image recognition processing. This is a diagram illustrating a method for identifying an occupant from multiple people.

[0010] Hereinafter, one embodiment of the passenger assistance system according to the present invention will be described in detail with reference to the drawings. First, the vehicle 2 equipped with the passenger assistance system 1 according to this embodiment will be described below. Figure 1 is a schematic configuration diagram of the vehicle 2 according to this embodiment.

[0011] [Description of Vehicle] Here, Vehicle 2 may be, for example, an automobile powered by an internal combustion engine (internal combustion engine automobile), an automobile powered by an electric motor (electric vehicle, fuel cell vehicle, etc.), or an automobile powered by both (hybrid vehicle). Furthermore, there is no restriction on the type of vehicle; it may be a regular passenger car, a large commercial truck, a bus, construction machinery, etc. Also, although the following description refers to it as a four-wheeled vehicle, it may also be a two-wheeled or three-wheeled vehicle.

[0012] Furthermore, Vehicle 2 may be a vehicle capable of not only manual driving based on the user's driving operations, but also assisted driving through automated driving assistance, where the vehicle drives automatically without user input. Alternatively, it may be a vehicle capable of only assisted driving through automated driving assistance. On the other hand, Vehicle 2 is not necessarily limited to a vehicle capable of assisted driving through automated driving assistance, and may be a vehicle capable of manual driving only.

[0013] As shown in Figure 1, the vehicle 2, which is a right-hand drive vehicle, is equipped with a driver's side front door 3R, a passenger side front door 3L, a driver's side rear door 4R, a passenger side rear door 4L, and a back door 5. Hereafter, when the front doors 3R, 3L, rear doors 4R, 4L, and back door 5 are described collectively, they may be referred to as "each door." Each door is, for example, a swing-type door. The vehicle 2 also has door lock devices 6A to 6E that control the locking of each door. The door lock devices 6A to 6E control the locking of the front doors 3R, 3L, rear doors 4R, 4L, and back door 5 in this order. More specifically, the door lock devices 6A to 6E are devices that switch each door between a locked state and an unlocked state. Furthermore, in addition to the door locking devices 6A to 6E, a door opening and closing device may be provided to automatically open and close each of the front doors 3R, 3L, rear doors 4R, 4L, and back door 5. The door opening and closing device may, for example, be equipped with a motor as a drive source and open and close each door by driving the motor.

[0014] Note that the configuration of vehicle 2 shown in Figure 1 is just one example. For example, vehicle 2 is not limited to a vehicle with the steering wheel on the right side; it could also be a vehicle with the steering wheel on the left side. Also, the doors are not limited to swing doors; other opening and closing mechanisms such as sliding doors can be used. Furthermore, the number of doors will vary depending on the vehicle type.

[0015] Furthermore, as shown in Figure 1, in addition to the aforementioned doors and door lock devices 6A to 6E, the vehicle 2 includes a front camera 7, a rear camera 8, and side cameras 9A and 9B (hereinafter collectively referred to as cameras) for imaging the area around the vehicle, a key radio wave transmission / reception unit 10, and a passenger assistance ECU (Electronic Control Unit) 11 that performs various calculations based on the input information. The passenger assistance system 1 includes the passenger assistance ECU 11 and other components.

[0016] The following describes each component of the vehicle 2 other than the doors and door locking devices 6A to 6E. First, the front camera 7 is an imaging device having a camera that uses a solid-state image sensor such as a CCD, and is installed, for example, above the front bumper of the vehicle 2 or behind the rearview mirror, with the optical axis facing forward in the direction of travel of the vehicle.

[0017] The rear camera 8 is an imaging device that also has a camera using a solid-state image sensor such as a CCD, and is mounted, for example, near the center above the license plate attached to the rear of the vehicle 2, with the optical axis facing the rear of the vehicle.

[0018] Furthermore, the side cameras 9A and 9B are imaging devices that also have cameras using solid-state image sensors such as CCDs, and are installed, for example, inside the left and right pillars of the vehicle 2, with the optical axis facing the side of the vehicle.

[0019] The ECU 11 then detects people around the vehicle by performing image recognition processing on the images captured by the front camera 7, rear camera 8, and side cameras 9A and 9B. If a person is present, the ECU also processes data to identify their speed, movement path, body orientation, body movements, and whether or not they are carrying luggage and what type it is. In addition to detecting people, each camera can also be used to capture images of the area around the vehicle for display on a vehicle-mounted display, or as a sensor to detect obstacles around the vehicle.

[0020] Furthermore, in addition to the camera mentioned above, vehicle 2 may also be equipped with sensors for detecting its surroundings, such as ultrasonic sensors, millimeter-wave sensors, laser sensors, etc. In addition to the camera that captures images outside the vehicle, it may also be equipped with a camera that captures images inside the vehicle.

[0021] On the other hand, the key radio wave transmitting / receiving unit 10 is a device that performs wireless communication with a portable device 16 carried by the occupant. The key radio wave transmitting / receiving unit 10 is equipped with antennas 15A to 15D for transmitting and receiving radio waves at multiple locations on the vehicle 2 (for example, at the four corners of the vehicle). However, the number and installation positions of the antennas 15A to 15D can be changed as appropriate. Here, the portable device 16 is, for example, a so-called electronic key. Alternatively, the portable device 16 may be a smartphone used in a digital key system (i.e., a portable terminal equipped with the function of an electronic key), or it may be another communication terminal capable of wireless communication with the vehicle 2.

[0022] The occupant support ECU 11 can detect the position of the portable device 16 using the key radio wave transmission / reception unit 10, that is, the position of the occupant assuming that the occupant of the vehicle is carrying the portable device 16. For example, by transmitting and receiving signals between multiple antennas 15A to 15D installed in different locations and the portable device 16, the distance from each antenna 15A to 15D to the portable device 16 can be calculated. Subsequently, the specific position of the portable device 16 (relative position to the vehicle) can be detected by triangulation using the calculated distances. In addition, in this embodiment, the position of the portable device 16 detected as described above is combined with the occupant detection using the aforementioned camera. For example, the position of the portable device 16 detected is used to identify an occupant from among multiple people, or to narrow down the image range for image recognition processing.

[0023] To briefly explain the operation of the key radio wave transmitting / receiving unit 10, the key radio wave transmitting / receiving unit 10 constantly emits weak radio waves into its surroundings to search for the target portable device 16 (the portable device 16 carried by the occupant). When the occupant carrying the target portable device 16 approaches the vehicle within a certain distance and the portable device 16 receives radio waves from the vehicle 2, the portable device 16 transmits a response signal, which the key radio wave transmitting / receiving unit 10 catches and compares with pre-registered authentication information. If the match is found, the unit detects that the occupant has approached the vehicle 2, and continues to detect the location of the portable device 16 until it can no longer transmit or receive signals.

[0024] Furthermore, the portable device 16 also has separate buttons for operating the door lock devices 6A to 6E and the door opening and closing device. Therefore, after the portable device 16 has been verified, in addition to the automatic unlocking and opening / closing of the doors as described later, it is also possible to unlock and open / close the doors by operating the buttons on the portable device 16.

[0025] On the other hand, the passenger assistance ECU 11 is an electronic control unit that performs various processes to assist passengers in boarding the vehicle 2. For example, it uses the key radio wave transmission / reception unit 10 to detect the position of the portable device 16 and activates the camera when the activation conditions are met. After the camera is activated, it uses the camera in addition to the position detection result of the portable device 16 to acquire the behavior of the passenger approaching the vehicle, and uses this information to read the passenger's intention to board the vehicle. If the passenger's intention to board is read, it controls the door lock devices 6A to 6E to automatically release the door lock of the door the passenger is trying to open, or further uses the door opening / closing device to automatically open or close the door. When acquiring the behavior of the passenger approaching the vehicle, it also performs image recognition processing on the images captured by the aforementioned front camera 7, rear camera 8, and side cameras 9A and 9B. The passenger assistance ECU 11 is connected to the aforementioned door lock devices 6A to 6E, front camera 7, rear camera 8, side cameras 9A and 9B, and key radio wave transmission / reception unit 10 via an in-vehicle network such as CAN. Furthermore, it is connected to various sensors mounted on vehicle 2, including GPS, vehicle speed sensor, acceleration sensor, gyro sensor, steering sensor, shift position sensor, and ultrasonic sensor. The detailed configuration of the occupant assistance ECU 11 will be described later.

[0026] In addition to the components shown in Figure 1, Vehicle 2 also has other basic components as a vehicle; however, only the configuration related to the control of the passenger assistance system and the control related to said configuration will be explained.

[0027] [Explanation of the Rider Support ECU] Next, we will explain in detail the Rider Support ECU 11, which is part of the Rider Support System 1 provided by the vehicle 2 described above. Figure 2 is a block diagram showing the configuration of the Rider Support System 1 according to this embodiment.

[0028] As shown in Figure 2, the passenger support ECU (Electronic Control Unit) 11 is an electronic control unit that controls the entire passenger support system 1. It includes a CPU 31 as an arithmetic unit and control device, a RAM 32 which is used as working memory when the CPU 31 performs various arithmetic operations and stores image recognition results, a ROM 33 which stores control programs as well as passenger support processing programs (see Figure 3) described later, and a flash memory 34 which stores programs read from the ROM 33. The passenger support ECU 11 also has various control units as processing algorithms. For example, the key position detection unit detects the position of the vehicle's key. The image acquisition unit acquires images captured by the vehicle's camera. The occupant identification unit identifies the occupants of the vehicle included in the image using the detection results from the key position detection unit. The passenger intention determination unit determines the occupant's intention to board based on the occupant's behavior identified by the occupant identification unit. The vehicle control unit performs vehicle control to assist boarding of the vehicle 2 when it determines that the occupant intends to board. In other words, the passenger assistance ECU 11 is an example of a key position detection unit, an image acquisition unit, a passenger identification unit, a passenger intention determination unit, and a vehicle control unit.

[0029] Furthermore, the occupant assistance ECU 11 is connected to various sensors 39 for detecting the vehicle's current position and behavior, including GPS, vehicle speed sensor, wheel speed sensor, acceleration sensor, gyro sensor, steering sensor, and shift position sensor.

[0030] Furthermore, the flash memory 34 includes a vehicle information DB 35 and a judgment information DB 36. The vehicle information DB 35 stores various information about the vehicle 2. For example, it stores the installation positions of cameras and sensors installed on the vehicle 2 (height from the ground, left-right position), detection axes (optical axis and field of view for cameras), overall length, vehicle width, wheelbase, minimum turning radius, and door handle positions. This information is entered in advance by the occupants or personnel from the vehicle manufacturer.

[0031] On the other hand, the judgment information DB36 stores information such as thresholds (judgment criteria) for determining whether or not an occupant intends to board vehicle 2, based on the occupant's behavior detected using a camera. The information in the judgment information DB36 may be pre-set by the vehicle manufacturer, for example, or the occupant may set it themselves after purchasing vehicle 2, taking usability into consideration.

[0032] [Explanation of the boarding support processing program] Next, the boarding support processing program executed by the boarding support ECU 11 in the boarding support system 1 having the above configuration will be explained with reference to Figure 3. Figure 3 is a flowchart of the boarding support processing program according to this embodiment. Here, the boarding support processing program is executed, for example, when the ACC power (accessory power supply) of the vehicle 2 is turned off and the vehicle is parked, and all doors are locked. The program determines whether or not the occupant intends to board the vehicle 2 based on the behavior of an occupant approaching the vehicle, and if it is determined that the occupant intends to board, it automatically unlocks the doors that the occupant is expected to open. However, the conditions for starting the execution of the boarding support processing program are not limited to the above conditions. For example, it may be executed to assist a new occupant to board when the ACC power of the vehicle 2 is on and an occupant has already boarded, or it may be executed when some of the doors are unlocked. The program shown in the flowchart in Figure 3 below is stored in the RAM 32 and ROM 33 of the boarding support ECU 11 and executed by the CPU 31.

[0033] First, in step 1 (hereinafter abbreviated as S), the CPU 31 uses the key radio wave transmitting / receiving unit 10 to poll (periodicly monitor) the portable device 16. Specifically, the key radio wave transmitting / receiving unit 10 constantly emits weak radio waves into its surroundings. When an occupant carrying the target portable device 16 approaches the vehicle within a certain distance and the portable device 16 receives radio waves from the vehicle 2, the portable device 16 emits a response signal, and the key radio wave transmitting / receiving unit 10 catches the reply signal from the portable device 16. In S1, the above-mentioned weak radio waves are emitted and the reply signal is caught.

[0034] Next, in S2, when the CPU 31 receives a reply signal from the portable device 16, it compares the received reply signal with pre-registered authentication information.

[0035] Subsequently, in S3, the CPU 31 determines whether the reply signal received from the portable device 16 matches the pre-registered authentication information as a result of the verification in S2. If a match is found (S3: YES), the CPU 31 recognizes that the portable device 16 that sent the reply is the portable device 16 held by the correct occupant who has the authority to unlock the doors of the vehicle 2, and then sets it as a target for linking. The linking continues until no more signals are received from the portable device 16, and the following processing from S4 onwards is performed on the linked portable device 16.

[0036] On the other hand, if no matching occurs (S3: NO), the system recognizes that the mobile device 16 that responded is owned by an occupant who does not have the authority to unlock the doors of vehicle 2, i.e., an occupant of another vehicle, and terminates without coordinating. The system then returns to S1 and continues polling.

[0037] In S4, the CPU 31 uses the key radio wave transmission / reception unit 10 to detect the position of the connected portable device 16. Assuming that the vehicle occupants are carrying the portable device 16, this also corresponds to detecting the occupants' positions. Here, the method for detecting the position of the portable device 16 involves transmitting and receiving signals between the portable device 16 and multiple antennas 15A to 15D installed at different locations on the vehicle. First, the distance from each antenna 15A to 15D to the portable device 16 is calculated. For example, this can be calculated from the time required for transmitting and receiving signals or the received signal strength. Subsequently, the specific position of the portable device 16 (relative position to the vehicle) can be detected by triangulation using the calculated distances. However, the method for detecting the position of the portable device 16 is not limited to the above example; for example, the portable device 16 itself may be equipped with a sensor to determine its current position. Furthermore, the detection of the position of the portable device 16 in S4 will be repeatedly performed at predetermined intervals until the connection with the portable device 16 is terminated, for example, when the portable device 16 moves to a distance where it is no longer possible to send or receive signals, thereby enabling real-time detection of the current position of the portable device 16. In particular, even after the camera activation conditions are met and imaging by the front camera 7, rear camera 8, and side cameras 9A and 9B begins, as described later, the detection of the current position of the portable device 16 will continue to be performed.

[0038] Subsequently, in S5, the CPU 31 determines whether the position of the portable device 16 detected in S4 is within a predetermined distance from the vehicle 2, that is, whether the occupant has approached within a predetermined distance from the vehicle 2. Here, as will be described later, in this embodiment, if the CPU 31 determines that the position of the portable device 16 is within a predetermined distance from the vehicle 2, it activates the camera and determines whether the occupant intends to board the vehicle. More specifically, as shown in Figure 4, when the occupant 41 carrying the portable device 16 moves in the direction of arrow 42 and approaches the vehicle 2, the position of the occupant 41 is detected using the key radio wave transmitting / receiving unit 10 (S4), and when it is determined that the occupant has approached within a predetermined distance X from the vehicle 2, the camera is activated, and monitoring of the occupant 41 by the camera begins. Monitoring of the occupant 41 by the camera continues as long as the occupant 41 is located in the area 43 within a predetermined distance X from the vehicle 2. Then, when the occupant 41 moves along arrow 44 and approaches the door, it is determined that the occupant intends to board, and the door is unlocked, as will be described later. On the other hand, if crew member 41 moves along arrow 45 and exits area 43, it will be determined that they have no intention to board, and the camera monitoring of crew member 41 will end (however, position detection of the handheld device 16 in S4 will continue).

[0039] The predetermined distance X should preferably be a distance such as 1m to 1.5m, at which point the camera can confirm the person's position and their intention to board the vehicle. At least passengers within area 43 can be captured by either the front camera 7, the rear camera 8, or the side cameras 9A and 9B.

[0040] If it is determined in S4 that the location of the portable device 16 is within a predetermined distance from the vehicle 2 (S5: YES), the process proceeds to S6. Conversely, if it is determined in S4 that the location of the portable device 16 is not within a predetermined distance from the vehicle 2 (S5: NO), the process returns to S4.

[0041] In S6, the CPU 31 activates the front camera 7, the rear camera 8, and the side cameras 9A and 9B. If the position of the occupants has been determined in advance, only the cameras whose imaging range includes the occupants' positions may be activated. In addition, to reduce the power consumption of the vehicle battery, the CPU 31 stops supplying power to each camera and turns off the power to each camera until S6 is executed. Alternatively, it may be kept in sleep mode (power-saving state). After S6, the CPU 31 starts supplying power to each camera, starts imaging the surroundings, and acquires imaging data from each camera.

[0042] Next, in S7, the CPU 31 obtains the location of the portable device 16 that was most recently detected using the key radio wave transmission / reception unit 10. Assuming that a crew member is carrying the portable device 16, the location of the portable device 16 also corresponds to the crew member's location.

[0043] Next, in S8, the CPU 31 converts the position coordinates of the portable device 16 detected using the key radio wave transmitting / receiving unit 10 into coordinates on the captured image 50. The position coordinates P of the portable device 16 detected using the key radio wave transmitting / receiving unit 10 are specified in a planar coordinate system parallel to the ground with the camera that captured the image (e.g., the side camera 9A) as the origin, as shown in Figure 5, or in a three-dimensional coordinate system that also includes the height direction. Therefore, the CPU 31 first performs a coordinate transformation to convert the position coordinates P of the portable device 16 into a two-dimensional coordinate system set for the captured image 50. For the coordinate transformation, parameters related to the camera (optical axis, installation position, field of view, etc.) stored in the vehicle information DB 35 are used. The transformed position coordinates P will be located near the feet of the occupant's image 51 included in the captured image 50.

[0044] Subsequently, in S9, the CPU 31 acquires real-time images captured by the front camera 7, rear camera 8, and side cameras 9A and 9B, and extracts images from the acquired images that correspond to the current position of the portable device 16 acquired in S7 (hereinafter referred to as the image processing area 52). Here, the image processing area 52 is the area of ​​the images captured by the cameras that is expected to contain an image of the occupant holding the portable device 16.

[0045] Specifically, the CPU 31 sets the image processing area 52 for the captured image 50 with reference to the position coordinate P after coordinate transformation in step S8. For example, in the example shown in Fig. 5, the image processing area 52 is set as a rectangular shape with a length X in the x-axis direction and a length Y in the y-axis direction. However, the shape is not necessarily required to be rectangular, and may also be a rhombus or an ellipse. In addition, the image processing area 52 is set to a range that is expected to include at least the whole body of the image of the occupant 41 whose foot is located at the position coordinate P. It is preferable that the values of X and Y are not fixed, but are changed according to the position coordinate P. Specifically, as the position of the occupant becomes farther from the vehicle, the image of the occupant 41 shown in the captured image 50 becomes smaller, so the values of X and Y are also decreased. Conversely, as the position of the occupant becomes closer to the vehicle, the image of the occupant 41 shown in the captured image 50 becomes larger, so the values of X and Y are also increased. In consideration of calculation errors of the position coordinate P, it is preferable that the image processing area 52 is set to be relatively large with a margin.

[0046] On the other hand, in step S9, instead of extracting the image processing area 52 from one captured image, the image processing area 52 may be extracted on a camera basis. That is, in the present embodiment, the vehicle 2 includes four cameras with different imaging ranges, namely a front camera 7, a rear camera 8, and side cameras 9A and 9B. Among the four captured images captured by the four cameras, the captured image including the position coordinate P of the portable device 16 detected by the key radio wave transmitting and receiving unit 10 may be extracted as the image processing area 52. For example, in the example shown in Fig. 5, the position coordinate P of the portable device 16 is included in the imaging range of the side camera 9A that captures an image of the right side of the vehicle 2, so the captured image captured by the side camera 9A is extracted as the image processing area 52.

[0047] Next, in step S10, the CPU 31 performs image recognition processing on the image processing area 52 extracted in step S9 among the real-time imaging data captured by the front camera 7, the rear camera 8, and the side cameras 9A and 9B. In particular, the processing after step S7 is performed in units of one frame of the imaging data acquired from each camera. Then, based on the result of the image recognition processing, the skeleton of a person (including at least the occupant, and possibly including a person other than the occupant) included in the captured image is detected.

[0048] Hereinafter, the process of S10 will be described. The CPU 31 executes a pre-stored skeleton detection program to detect a human skeleton included in a captured image. In the skeleton detection program, for example, a posture estimation AI is used to estimate and detect feature points such as a person's face, waist, joints (wrist, ankle, knee, elbow, shoulder, neck) from above clothes, and a human skeleton can be detected by connecting these feature points with line segments. For example, FIG. 6 shows an example in which skeleton detection is performed on a human image 51 included in the image processing area 52 of the captured image 50. As shown in FIG. 6, in the human image 51, the face, waist, joints (wrist, ankle, knee, elbow, shoulder, neck) and the like are specified as feature points 53, and the skeleton of the occupant is specified by line segments 54 connecting these feature points.

[0049] Furthermore, in S10, the CPU 31 specifies the position Q of the person's foot on the captured image 50 based on the human skeleton detected using the captured image. Note that the foot position Q may be an intermediate position between the left and right ankle positions, or may be the ankle position of either one (for example, the one closer to the vehicle). Further, if the position of the toe of the foot can also be specified by skeleton detection, the position of the toe instead of the ankle position may be acquired as the position of the occupant's foot. The foot position Q corresponds to the position of the person included in the captured image 50. Note that the process of S10 is performed for each person included in the image processing area 52. That is, when a plurality of persons are included in the image processing area 52, the above-described skeleton detection and specification of the foot position Q are performed for each person.

[0050] Subsequently, in S11, the CPU 31 confirms consistency (matching) between the position of the portable device 16 detected using the key radio transmitting and receiving unit 10 acquired in S7 and the processing result of the image recognition processing in S10.

[0051] The following describes the process in S11. First, if the image processing area 52 contains multiple people, the CPU 31 uses the foot position Q identified in S10 and the camera parameters (optical axis, installation position, field of view, etc.) stored in the vehicle information DB 35 to perform a coordinate transformation to identify the coordinates of the foot position Q identified on the captured image in a planar coordinate system parallel to the ground with the camera that captured the image (e.g., the side camera 9A) as the origin, or in a three-dimensional coordinate system that also includes the height direction. Then, the CPU 31 compares the foot position Q after the coordinate transformation with the position coordinates P of the portable device 16 detected using the key radio wave transmitting / receiving unit 10, and identifies the person whose foot position Q is closest to the position coordinates P. It should be assumed that the position coordinates P are identified in the same coordinate system as the coordinates of the foot position Q by performing the coordinate transformation in S8. After that, it checks whether the foot position Q of the identified person and the position coordinates P of the portable device 16 are consistent (whether the difference is within a predetermined distance), and if they are consistent, the person is confirmed as an occupant of the vehicle 2. For example, as shown in Figure 7, if the image processing area 52 contains both person 55 and person 56, the distance L1 from the position Q1 of person 55's feet to the position coordinate P of the portable device 16 is compared with the distance L2 from the position Q2 of person 56's feet to the position coordinate P of the portable device 16. In the example shown in Figure 7, distance L1 is shorter than distance L2, meaning that the position Q1 of the feet is closer to the position coordinate P of the portable device 16. Therefore, person 55, whose position is Q1, is identified as an occupant of vehicle 2. However, if distance L1 is greater than or equal to a predetermined distance, person 55 is not identified as an occupant of vehicle 2. In that case, the occupant cannot be identified, and the process in S12 is not performed, returning to S7.

[0052] On the other hand, if there is only one person in the image processing area 52, the coordinate transformation of the foot position Q identified on the captured image is performed on that person, and the foot position Q after the coordinate transformation is compared with the position coordinate P of the portable device 16 detected using the key radio wave transmitting / receiving unit 10. Then, it is checked whether the foot position Q after the coordinate transformation and the position coordinate P of the portable device 16 are consistent (whether the difference is within a predetermined distance), and if they are consistent, that person is confirmed as an occupant. If the difference is greater than the predetermined distance, the occupant of the vehicle 2 cannot be identified, and the process in S12 is not performed, and the system returns to S7.

[0053] By performing the processes S9 to S11 described above, if the captured image 50 taken by the camera includes the occupants of vehicle 2, the occupants of vehicle 2 will be identified. Then, the following process S12 is performed on the identified occupants. After the occupants of vehicle 2 are identified in the process S11, only processes S10, S12, and S13 may be repeatedly executed until the identified occupants disappear from the captured image taken by the camera.

[0054] In S12, the CPU 31 detects the behavior of the occupant identified in S11. For example, it detects the position Q of the feet of a person identified as an occupant, identified in S10, as the occupant's position, detects the line connecting the history of the foot position Q's past positions as the occupant's movement path (trajectory), detects the amount of change in the foot position Q per unit time as the occupant's speed, and identifies the orientation of the occupant's body from the skeleton detected in S10. The behavior of the occupant to be detected includes, for example, position, movement path (trajectory), speed, and body orientation, but other behaviors may also be included in the detection targets. For example, the presence and type of luggage carried by the occupant may also be detected based on the results of the image recognition processing. Furthermore, since the image recognition processing includes skeleton detection as described above, it is possible to more specifically identify the posture and angle of the limbs. The process then proceeds to S13.

[0055] In S13, the CPU 31 determines, based on the occupant's behavior acquired in S12, whether the occupant has the intention to board the vehicle 2 (which can also be described as the intention to open the door of the vehicle 2). Specifically, it is determined that the occupant has the intention to board if the following conditions are met: [Conditions] - The occupant's current position is within a predetermined distance (for example, 1 m) from the position of any of the doors of the vehicle 2 (front doors 3R, 3L, rear doors 4R, 4L, back door 5). - The occupant's body is facing the direction of the door that has been determined to be within the predetermined distance (the door is in the occupant's direction of travel). - The occupant's movement path (trajectory) is a roughly straight line towards the door that has been determined to be within the predetermined distance (it is not a movement path that involves staying around the vehicle or loitering).

[0056] However, the above conditions are just examples, and the intention to board may be determined by other conditions. For example, the position of the occupant's hands may be identified from the image recognition results, and conditions may be set such as the occupant's hands being within a predetermined distance from the door handle or the hands facing the direction of the door handle.

[0057] Then, based on the occupant's behavior obtained in S12, if it is determined that the occupant intends to board vehicle 2 (S13: YES), the process proceeds to S14. On the other hand, if it is determined that the occupant does not intend to board vehicle 2 (S13: NO), the process returns to S7.

[0058] In S14, the CPU 31 identifies the door to be unlocked. Specifically, the door that is determined to be within a predetermined distance from the occupant under the above conditions becomes the door to be unlocked. Note that the doors to be unlocked are not limited to the doors that are opened when getting in (e.g., front doors 3R, 3L, rear doors 4R, 4L), but also include the doors that are opened when loading luggage (e.g., the back door 5). Furthermore, the door to be unlocked may be identified based on whether or not the occupant is carrying luggage and the type of luggage. For example, if the occupant is carrying large luggage such as a trunk, the back door 5 may be identified as the door to be unlocked, and if the occupant is carrying a child, the rear doors 4R, 4L may be identified as the doors to be unlocked.

[0059] Subsequently, in S15, the CPU 31 operates the door lock devices 6A to 6E corresponding to the door identified in S14, and unlocks the corresponding door. In addition to unlocking the door, the system may also automatically open the door afterward. Furthermore, the system may also provide additional assistance for passengers to board the vehicle, such as turning on the interior lights or starting the engine.

[0060] As a result, the occupant can automatically unlock the door simply by moving near the door they wish to open, without having to operate the portable device 16 or the door handle. If necessary, the door can also be automatically opened and the engine started. In addition to unlocking the doors using the above-mentioned passenger assistance program, the occupant can, of course, also manually unlock the doors as before.

[0061] As described in detail above, according to the passenger boarding support system 1 and the computer program executed by the passenger boarding support system 1 according to this embodiment, the position of the portable device 16, which corresponds to the key of the vehicle 2, is detected (S4), an image 50 captured by a camera installed in the vehicle 2 is acquired (S6), the occupants of the vehicle 2 included in the image 50 are identified using the detection result of the position of the portable device 16 (S9 to S11), the occupants' intention to board is determined based on the behavior of the identified occupants (S13), and if it is determined that the occupants intend to board, vehicle control is performed to assist them in boarding the vehicle (S15). By combining the detection result of the position of the portable device 16 with the determination of the occupants' intention to board using the camera, it becomes possible to accurately grasp the occupants' intention to board as early as possible. As a result, it becomes possible to perform vehicle control to assist them in boarding the vehicle without delay. Furthermore, from the image captured by the camera, the image of the image processing area 52, which is the area corresponding to the position of the portable device 16, is extracted (S9), and the occupants included in the image are identified by performing image recognition processing on the extracted image (S10, S11). This reduces the processing load of the image recognition process for identifying occupants and detecting their behavior, and shortens the processing time, by narrowing down the area to be processed for image recognition in advance. Furthermore, by performing image recognition processing on the captured image, the position of people included in the captured image is detected (S10), and the position of the portable device 16 is matched with the position of the people (S11), thereby identifying the occupants included in the captured image. This prevents the misidentification of other people as occupants by combining the image recognition results of the camera with the detection results of the position of the portable device 16. In addition, when matching the position of the portable device 16 with the position of the people, the position coordinates of the person's feet, which are identified on the captured image by performing image recognition processing on the captured image, are transformed to be identified in a planar coordinate system parallel to the ground with the camera as the origin, or in a three-dimensional coordinate system that also includes the height direction. The position coordinates of the person's feet after the coordinate transformation are compared with the coordinates of the portable device 16, which are identified in the same coordinate system as the position of the person's feet, to confirm whether they are matched (S11).This makes it possible to accurately compare the location of the person identified by the image recognition result with the location detection result of the portable device 16 using the same coordinate system and confirm whether they are consistent. Furthermore, from the captured images taken by the camera, the image of the image processing area 52 corresponding to the location of the key is extracted (S9), and if multiple people are included in the extracted image, the location of each of the multiple people is detected by performing image recognition processing on the image (S10), and the person closest to the location of the key is identified as the passenger (S11). As a result, even if there are multiple people who are candidates for passengers, by using the detection result of the location of the portable device 16 in combination, it becomes possible to accurately grasp the passenger's intention to board the vehicle as early as possible. Consequently, it becomes possible to perform vehicle control to assist in boarding the vehicle without delay.

[0062] It should be noted that the present invention is not limited to the above embodiments, and various improvements and modifications are possible without departing from the spirit of the invention. For example, in this embodiment, as a control to assist in boarding the vehicle when the intention of the occupant to board is confirmed, a control to release the door lock is performed (S15), but it may also be a control to open the door (open and close the door), or both, or it may be a control to turn on the engine or turn on the interior lights.

[0063] In this embodiment, the camera is activated when it is determined that the portable device 16 is within a predetermined distance from the vehicle 2 (S5: YES), but the camera activation conditions may be other conditions. For example, the conditions may be that the portable device 16 is within a predetermined distance from the vehicle 2 AND the portable device 16 is moving in the direction of the vehicle 2.

[0064] Furthermore, in this embodiment, the doors that are subject to unlocking by the above-described boarding assistance processing program are not limited to doors that are opened when boarding (e.g., front doors 3R, 3L, rear doors 4R, 4L), but also include doors that are opened when loading luggage (e.g., back door 5). However, it is also possible to target only the doors that are opened when boarding. Alternatively, it is also possible to target only a specific door (e.g., the driver's side door).

[0065] In this embodiment, the image processing area 52 to be subject to image recognition processing is narrowed down from the images captured by the front camera 7, rear camera 8, and side cameras 9A and 9B (S9), but the camera to be activated may also be narrowed down. That is, in this embodiment, the vehicle 2 is equipped with four cameras, the front camera 7, rear camera 8, and side cameras 9A and 9B, each with different imaging ranges, but only the camera whose imaging range includes the position coordinates P of the portable device 16 detected by the key radio wave transmitting / receiving unit 10 may be activated, and the processing from S10 onwards may be performed.

[0066] Furthermore, the execution order of each step in the passenger assistance processing program shown in Figure 3 is just an example, and the execution order can be changed as appropriate. For example, the authentication of the portable device 16 in S2 may be performed after the processing in S4. It is also possible to omit some of the steps included in the passenger assistance processing program. For example, the processing in S8 and S9 may be omitted. That is, the processing from S10 onwards may be performed without narrowing down the image processing area 52. Even in that case, by matching the detection result of the position of the portable device 16 with the image recognition result in S11, it is possible to identify the passenger from among the captured images even if multiple people are included in the image.

[0067] Furthermore, in this embodiment, the passenger assistance ECU 11 of the passenger assistance system 1 executes the processing of the passenger assistance processing program (Figure 3), but the execution entity can be changed as appropriate. For example, it may be configured so that the vehicle control ECU, the control unit of the navigation system, or other in-vehicle devices execute the processing.

[0068] [Summary of this embodiment] This embodiment comprises at least the following configurations: a key position detection unit (11) that detects the position of the key (16) of the vehicle (2); an image acquisition unit (11) that acquires an image (50) captured by an imaging device (7, 8, 9A, 9B) provided by the vehicle; an occupant identification unit (11) that identifies the occupants (41) of the vehicle included in the image using the detection result of the key position detection unit; an occupant intention determination unit (11) that determines the occupant's intention to board the vehicle based on the behavior of the occupant identified by the occupant identification unit; and a vehicle control unit (11) that performs vehicle control to assist the occupant in boarding the vehicle when it is determined that the occupant intends to board the vehicle.

[0069] This configuration allows for the accurate and earliest possible determination of a passenger's intention to board a vehicle by combining the detection results of the key's position with the determination of the passenger's intention to board using an imaging device. As a result, vehicle control can be performed to assist passengers in boarding the vehicle without delay.

[0070] Furthermore, in this embodiment, it is preferable that the occupant identification unit (11) extracts an image from the captured image (50) acquired by the image acquisition unit (11) that corresponds to the position of the key detected by the key position detection unit (11), and then performs image recognition processing on the extracted image to identify the occupant included in the image.

[0071] This configuration allows for a reduction in the processing load of image recognition for identifying occupants and detecting their behavior, and also shortens processing time, by narrowing down the area to be processed beforehand.

[0072] Furthermore, in this embodiment, the occupant identification unit (11) preferably identifies the occupant included in the captured image (50) by performing image recognition processing on the captured image (50), and by matching the position of the key (16) detected by the key position detection unit (11) with the position of the person.

[0073] This configuration prevents the system from mistakenly identifying other people as occupants by combining the image recognition results from the imaging device with the detection results for the key's position.

[0074] Furthermore, in this embodiment, when the occupant identification unit (11) performs image recognition processing on the captured image (50) to determine the position coordinates of the person's feet, it is preferable to perform a coordinate transformation to determine the position coordinates of the person's feet, which are identified on the captured image, in a planar coordinate system parallel to the ground with the imaging device as the origin, or in a three-dimensional coordinate system that also includes the height direction. The occupant identification unit (11) then checks whether the positions match by comparing the position coordinates of the person's feet after the coordinate transformation with the coordinates of the key position detected by the key position detection unit (11) in the same coordinate system as the position coordinates of the person's feet.

[0075] This configuration allows for accurate comparison of the location of a person identified by image recognition and the location of a key, using the same coordinate system, to verify whether they are consistent or not.

[0076] Furthermore, in this embodiment, the occupant identification unit (11) extracts an image from the image (50) acquired by the image acquisition unit (11) that corresponds to the position of the key detected by the key position detection unit (11). If multiple people are included in the extracted image, the unit performs image recognition processing on the image to detect the positions of each of the multiple people, and identifies the person closest to the position of the key detected by the key position detection unit (11) as the occupant.

[0077] With this configuration, even if there are multiple potential passengers, the system can accurately and quickly determine the passenger's intention to board by combining the detection results of the key's position. As a result, it becomes possible to control the vehicle to assist in boarding without delay.

[0078] 1...Riding assistance system, 2...Vehicle, 3R, 3L...Front door, 4R, 4L...Rear door, 5...Back door, 6A-6E...Door lock device, 7...Front camera (imaging device), 8...Rear camera (imaging device), 9A, 9B...Side camera (imaging device), 10...Key radio wave transmission / reception unit, 11...Riding assistance ECU (Example of key position detection unit, image acquisition unit, occupant identification unit, occupant intention determination unit, vehicle control unit), 16...Portable device (key), 31...CPU, 41...Occupant, 50...Image captured, 52...Image processing area

Claims

1. A passenger boarding support system comprising: a key position detection unit for detecting the position of a vehicle key; an image acquisition unit for acquiring an image captured by an imaging device provided by the vehicle; an occupant identification unit for identifying an occupant of the vehicle included in the image captured using the detection result of the key position detection unit; an occupant boarding intention determination unit for determining the occupant's intention to board based on the behavior of the occupant identified by the occupant boarding intention unit; and a vehicle control unit for performing vehicle control to assist the occupant in boarding the vehicle when it is determined that the occupant intends to board.

2. The passenger identification unit extracts an image of the area corresponding to the key position detected by the key position detection unit from the image acquired by the image acquisition unit, and identifies the passenger included in the image by performing image recognition processing on the extracted image.

3. The passenger identification unit detects the position of a person included in the captured image by performing image recognition processing on the captured image, and identifies the passenger included in the captured image by matching the position of the key detected by the key position detection unit with the position of the person, thereby enabling the passenger to be identified in the captured image, according to claim 1.

4. The passenger identification unit, when matching the position of the key with the position of the person, performs a coordinate transformation to identify the position coordinates of the person's feet, which are identified on the captured image by performing image recognition processing on the captured image, in a planar coordinate system parallel to the ground with the imaging device as the origin, or in a three-dimensional coordinate system including the height direction, and confirms whether they are matched by comparing the position coordinates of the person's feet after the coordinate transformation with the coordinates of the key position identified by the key position detection unit in the same coordinate system as the position coordinates of the person's feet.

5. The passenger identification unit extracts an image of an area corresponding to the position of the key detected by the key position detection unit from the image acquired by the image acquisition unit, and if multiple people are included in the extracted image, it performs image recognition processing on the image to detect the positions of each of the multiple people, and identifies the person closest to the position of the key detected by the key position detection unit as the passenger, according to claim 3.

Images