Seat position determination system, seat position determination device, and seat position determination method
Patent Information
- Application Number
- PCT/JP2025/012574
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2025-03-27
- Publication Date
- 2026-10-01
Smart Images
Figure JP2025012574_01102026_PF_FP_ABST
Abstract
Description
Seat position determination system, seat position determination apparatus, and seat position determination method
[0001] The present invention relates to a seat position determination system, a seat position determination apparatus, and a seat position determination method.
[0002] Conventionally, there is known a technique for determining a seat position in order to appropriately operate an occupant protection device when necessary, such as when a vehicle collides with an obstacle or the like.
[0003] For example, an airbag deployment timing control device disclosed in Patent Document 1 includes an impact sensor, a collision determination means, a seat position sensor, and an airbag deployment timing correction means. The impact sensor detects an impact applied to a vehicle. The collision determination means determines a collision from a detection output signal of the impact sensor. The seat position sensor detects a seat adjustment setting state of an occupant. The airbag deployment timing correction means performs timing correction on an output signal of the collision determination means based on a detection signal of the seat position sensor. An airbag deployment device is driven by the signal corrected by the airbag deployment timing correction means.
[0004] An occupant detection device disclosed in Patent Document 2 includes a seat belt having a row of markers, a camera that captures an image of the row of markers on the seat belt, and an estimation unit that estimates a seating state of an occupant wearing the seat belt based on an arrangement of the row of markers on an image captured by the camera.
[0005] Japanese Unexamined Patent Publication No. Hei 3-159838, Japanese Unexamined Patent Publication No. 2006-8110
[0006] However, in the configuration in which the seat adjustment setting state of the occupant is detected only by the seat position sensor as in Patent Document 1, the detection accuracy may decrease depending on the detection accuracy of the sensor mounted as the seat position sensor. Further, in the configuration in which a row of markers is detected from an image captured by a camera as in Patent Document 2, if the occupant does not correctly wear the seat belt, the seating state of the occupant may not be correctly detected in some cases.
[0007] This invention has been made in view of the circumstances described above, and aims to provide a seat position determination system, a seat position determination device, and a seat position determination method that improve the accuracy of seat position determination, with the aim of improving the safety of occupants. Furthermore, it aims to further improve traffic safety and contribute to the development of a sustainable transportation system.
[0008] The seat position determination system of the present disclosure comprises: an imaging device for imaging the inside of a moving body; and a control device for determining the seat position of a seat provided on the moving body, wherein the control device comprises: a determination unit for determining which seat among the seats provided on the moving body is to be used for determining the seat position; a detection unit for detecting a buckle member, a waist webbing member, and a shoulder webbing member included in a seat belt device from an image captured by the imaging device; a selection unit for selecting at least one member from the buckle member, the waist webbing member, and the shoulder webbing member to be used for determining the seat position based on the detection result of the detection unit; and a determination unit for determining the seat position based on the detection result of the detection unit for the at least one member selected by the selection unit.
[0009] According to the seat position determination system of this disclosure, the detection unit selects at least one member to be used for determining the seat position based on the detection results of the buckle member, waist webbing member, and shoulder webbing member detected from the captured image, and determines the seat position based on the detection results of the detection unit of the selected member. Therefore, the influence of the occupant's physique, posture, etc., when seated in the seat can be reduced, and the accuracy of seat position determination can be improved.
[0010] Figure 1 is a diagram showing the system configuration of the seat position determination system of Embodiment 1. Figure 2 is a side view of the driver's seat. Figure 3 is a schematic configuration diagram of the seat belt device viewed from the front. Figure 4 is a diagram illustrating a method for determining the seat position based on the position of the buckle member in a captured image. Figure 5 is a diagram illustrating a method for determining the seat position based on the position of the lumbar webbing member in a captured image. Figure 6 is a diagram illustrating a method for determining the seat position based on the position of the shoulder webbing member in a captured image. Figure 7 is a diagram showing a state in which some members of the seat belt device are obscured by an obstacle. Figure 8 is a flowchart showing the operation of the control device. Figure 9 is a flowchart showing the operation of a modified control device. Figure 10 is a flowchart showing other operations of the modified control device. Figure 11 is a diagram showing the system configuration of the seat position determination system of Embodiment 2.
[0011] Embodiments of this disclosure will be described below with reference to the drawings.
[0012] [1. Embodiment 1] [1-1. Configuration of the Seat Position Determination System] Figure 1 is a diagram showing the system configuration of the seat position determination system 1A of Embodiment 1. The system configuration of the seat position determination system 1A will be described with reference to Figure 1. The seat position determination system 1A comprises a shooting device 30, a seating detection sensor 40, a buckle switch 246, a touch panel 50, a vehicle ECU (Electronic Control Unit) 60, an occupant protection device 70, and a control device 100, and is mounted on a vehicle 10.
[0013] The camera 30, seat detection sensor 40, buckle switch 246, touch panel 50, vehicle ECU 60, occupant protection device 70, and control device 100 are connected to each other via an in-vehicle network (not shown) to enable data communication. The in-vehicle network uses a communication bus that supports communication standards such as CAN (Control Area Network), CAN-FD (CAN with Flexible Data Rate), and Ethernet. Ethernet is a registered trademark.
[0014] Figure 2 shows the mounting positions of the camera 30 and the seat detection sensor 40 in the vehicle 10. The camera 30 is, for example, a digital camera that photographs the interior of the vehicle 10. When an occupant 20 is seated in the vehicle 10, the camera 30 is installed, for example, in front of the roof of the vehicle 10 or on top of the windshield, so as to be able to photograph the upper body of the occupant 20. The camera 30 photographs the interior of the vehicle 10, generates an image, and outputs the generated image to the control device 100. The image captured by the camera 30 may be a moving image or a still image. Furthermore, while Embodiment 1 and Embodiment 2, which will be described later, show examples where one camera 30 is installed in the vehicle 10, multiple camera 30 may be installed in the vehicle 10. For example, a separate camera 30 may be provided to photograph the occupants 20 seated in the driver's seat and passenger seat of the vehicle 10, and another camera 30 may be provided to photograph the occupants 20 seated in the rear seat of the vehicle 10.
[0015] The seating detection sensor 40 is a pressure-sensitive sensor that detects the pressure applied to the seat by the weight of a person sitting on it. The seating detection sensor 40 is installed on each of the multiple seats in the vehicle 10. The seating detection sensor 40 detects the pressure applied to the seat and outputs sensor data indicating the detection result to the control device 100.
[0016] In Embodiment 1 and Embodiment 2, which will be described later, a configuration is described in which a seating detection sensor 40 is provided in the vehicle 10 to detect the seat in which an occupant 20 is seated. However, a configuration in which the seat in which an occupant 20 is seated may also be used, based on the image captured by the camera 30. In the case of the driver's seat, a heart rate sensor may be provided on the steering wheel of the vehicle 10, and it may be determined whether or not an occupant 20 is seated in the driver's seat of the vehicle 10 based on the sensor data detected by this heart rate sensor.
[0017] Next, the configuration of the seat belt device 250 will be described with reference to Figure 3. Figure 3 shows the driver's seat 2 as an example of a seat provided in the vehicle 10. The seat 2 comprises a seat cushion 211 that mainly supports the buttocks of the occupant 20, a seat back 212 that mainly supports the back of the occupant 20, and a headrest 213 that mainly supports the head of the occupant 20.
[0018] The seat cushion 211 is installed on the vehicle body floor of the vehicle 10 so as to be movable in the longitudinal direction of the vehicle body via a seat slide adjuster 220. The seat back 212 is provided so as to be tiltable in the longitudinal direction relative to the seat cushion 211 via a reclining adjuster. The reclining adjuster is not shown in the illustration. Both the seat slide adjuster 220 and the reclining adjuster have the same configuration as conventional ones and are well known technology. The seat slide adjuster 220 makes it possible to adjust the longitudinal position of the seat cushion 211 in the vehicle body, and the reclining adjuster makes it possible to adjust the tilt angle of the seat back 212.
[0019] The seat belt device 250 includes a seat belt 230 that restrains the occupant 20 to the seat 2, a lower anchor 241, a buckle member 243, a tongue plate 245, a shoulder anchor 247, and a retractor 240.
[0020] The seat belt 230 comprises a waist webbing member 231, which is a belt that crosses the wearer's waist and secures the wearer's pelvis, and a shoulder webbing member 233, which is a belt that secures the wearer's chest diagonally from the shoulders to the waist.
[0021] One end of the seat belt 230 is secured to the vehicle floor via a lower anchor 241 located on the outside of the seat 2, and the other end of the seat belt 230 is connected to the retractor 240 via a shoulder anchor 247 which is installed near the top of the seat back 212 and is vertically movable along the center pillar 270.
[0022] Between the lower anchor 241 and the shoulder anchor 247, the seat belt 230 is inserted through the tongue plate 245. The tongue plate 245 is configured to be detachable from a buckle member 243 attached to the vehicle floor on the interior side of the seat 2.
[0023] The seat belt device 250 includes a buckle switch 246. This buckle switch 246 outputs an ON signal to the control device 100 when the tongue plate 245 is engaged with the buckle member 243, that is, when the seat belt 230 is fastened. It also outputs an OFF signal to the control device 100 when the tongue plate 245 is released from the buckle member 243, that is, when the seat belt is not fastened. The ON signal and the OFF signal are collectively referred to as notification signals.
[0024] [1-2. Configuration of the Control Device] Returning to Figure 1, we will continue to describe the devices that constitute the seat position determination system 1A. The touch panel 50 comprises a display panel and a touch sensor. The display panel and touch sensor are not shown in the diagram. The display panel is made of a liquid crystal panel or an organic EL (Electro-Luminescence) panel. An image is displayed on the display panel under the control of the control device 100. The touch sensor detects touch operations on the display panel and outputs position information indicating the location of the detected touch operation to the control device 100.
[0025] The vehicle ECU 60 is an electronic control unit that controls the driving of the vehicle 10, and is a higher-level control unit than the control unit 100. For example, when the IG (IGNITION) switch of the vehicle 10 is turned on, the vehicle ECU 60 notifies the control unit 100 that the IG switch has been turned on.
[0026] The occupant protection device 70 includes an airbag and an inflator that supplies inflation gas to the airbag, and deploys the airbag to protect the occupant 20 when the vehicle 10 collides with an obstacle.
[0027] Next, the configuration of the control device 100 will be described. The control device 100 is a computer device such as an ECU that includes an input / output interface 110, a storage unit 120, and a processor 130. Hereafter, the interface will also be abbreviated as I / F.
[0028] The input / output interface 110 is an interface connected to an in-vehicle network (not shown). The input / output interface 110 is connected to the camera 30, seat detection sensor 40, buckle switch 246, touch panel 50, vehicle ECU 60, and occupant protection device 70 via the in-vehicle network.
[0029] The storage unit 120 includes at least non-volatile semiconductor memory. The storage unit 120 may also be configured to include volatile semiconductor memory or auxiliary storage devices. The storage unit 120 stores the control program 121 executed by the processor 130 and the reference position 123. The reference position 123 will be described later. The storage unit 120 also temporarily stores the images captured by the imaging device 30.
[0030] The processor 130 includes a determination unit 131, a detection unit 133, a selection unit 135, and a judgment unit 137 as functional units. These functional units represent the functions realized when the processor 130 executes the control program 121.
[0031] The determination unit 131 determines which seats are to be determined for seat position determination. The determination unit 131 receives sensor data output by the seating detection sensor 40 and notification signals output by the buckle switch 246. Based on the input from the seating detection sensor 40 and the buckle switch 246, the determination unit 131 determines which seats are to be determined for seat position determination. The determination unit 131 determines which seats are to be determined for seat position determination if the seating detection sensor 40 detects that an occupant 20 is seated and the buckle switch 246 detects that the seat belt 230 is being worn. Alternatively, the determination unit 131 may use the camera 30 as a sensor and determine which seats are to be determined for seat position determination based on the images captured by the camera 30.
[0032] The detection unit 133 acquires the captured image from the storage unit 120 by reading the image captured by the imaging device 30. The detection unit 133 analyzes the acquired image and detects the buckle member 243, waist webbing member 231, and shoulder webbing member 233 that are captured in the image. The detection unit 133 stores the detection results in the storage unit 120.
[0033] The selection unit 135 determines whether the determination unit 137 can determine the seat position based on the detection results of the detection unit 133 of the buckle member 243, the waist webbing member 231, and the shoulder webbing member 233. The selection unit 135 determines that the determination unit 137 can determine the seat position if the detection unit 133 detects at least one of the buckle member 243, the waist webbing member 231, and the shoulder webbing member 233. The selection unit 135 determines that the determination unit 137 cannot determine the seat position if the detection unit 133 fails to detect the buckle member 243, the waist webbing member 231, and the shoulder webbing member 233. If the selection unit 135 determines that the determination unit 137 cannot determine the seat position, it causes the detection unit 133 to reacquire the captured image and perform image analysis again.
[0034] Next, when the selection unit 135 determines that the determination unit 137 can determine the seat position, it selects the member to be used for determining the seat position based on the detection result of the detection unit 133. The selection unit 135 selects the member to be used for determining the seat position from the buckle member 243, the waist webbing member 231, and the shoulder webbing member 233. The selection unit 135 selects the member detected by the detection unit 133 through image analysis as the member to be used for determining the seat position.
[0035] Furthermore, the selection unit 135 may use the detection unit 133's detection of the buckle member 243 from the captured image as a condition for determining whether the seat position can be determined by the determination unit 137. The buckle member 243 is less affected by the occupant's physique, so the seat position can be determined with high accuracy. For this reason, the selection unit 135 may determine whether the seat position can be determined by the determination unit 137 on the condition that the detection unit 133 has detected the buckle member 243.
[0036] Furthermore, if the determination unit 131 determines multiple seats to be determined, the selection unit 135 may use the detection unit 133 detecting at least two or more components from the captured image as a condition for determining whether the seat position can be determined by the determination unit 137. For example, if the driver's seat and passenger seat, or two side-by-side seats in the rear seats, are determined to be determined, the selection unit 135 may use the detection unit 133 detecting at least two or more components from the captured image as a condition for determining whether the seat position can be determined by the determination unit 137. When there are multiple seats to be determined, if multiple components of adjacent seats are in close proximity, the positions of the buckle member 243, waist webbing member 231, and shoulder webbing member 233 may be misdetermined. For this reason, when there are multiple seats to be determined, the selection unit 135 may use the detection unit 133 detecting at least two or more components as a condition for determining whether the seat position can be determined.
[0037] The determination unit 137 determines the seat position based on the detection result of the detection unit 133 of at least one member selected by the selection unit 135. The seat position determined by the determination unit 137 is the position of the seat cushion 211 in the longitudinal direction of the vehicle body. In this embodiment, the case in which the seat position includes three positions: a forward position, a central position, and a rearward position is described, but a configuration in which multiple seat positions can be set is also possible.
[0038] Figure 4 is a diagram illustrating a method for determining the seat position based on the position of the buckle member 243 in the captured image 200. When the selection unit 135 selects the buckle member 243, the determination unit 137 determines the seat position based on the distance between the position of the buckle member 243 detected by the detection unit 133 from the captured image 200 and a pre-registered reference position 123. The position of the buckle member 243 detected by the detection unit 133 from the captured image 200 may be the central position in the height direction of the member detected by the detection unit 133 as the buckle member 243, or it may be the highest position in the height direction of the member detected by the detection unit 133 as the buckle member 243. The height direction refers to the vertical direction in the drawing view of the captured image 200 shown in Figure 4.
[0039] The reference position 123 is information that is registered in the control device 100 in advance. Alternatively, the reference position 123 may be set by a registration process that accepts operation input from the occupant 20. For example, the occupant 20 inputs seat identification information and seat position information of the seat in which the occupant 20 is seated via the touch panel 50. The seat identification information includes, for example, the driver's seat, passenger seat, rear seat on the driver's side, and rear seat on the passenger side. The seat position information indicates the position of the seat in the longitudinal direction of the vehicle body and includes the forward position, center position, and rear position.
[0040] When the occupant 20 inputs seat identification information and seat position information, for example, presses the registration button for the reference position 123 displayed on the touch panel 50. When the registration button is pressed, the control device 100 causes the camera 30 to take a picture and acquires a captured image 200 from the camera 30. The control device 100 detects the buckle member 243, waist webbing member 231, and shoulder webbing member 233 from the acquired captured image 200 and registers the positions of the detected buckle member 243, waist webbing member 231, and shoulder webbing member 233 as the reference position 123. At this time, the control device 100 associates the detected reference position 123, seat identification information, and seat position information and registers them as the reference position 123 in the storage unit 120.
[0041] The determination unit 137 determines that the seat position of the seat to be determined is the rear position if the difference in distance between the position of the buckle member 243 detected from the captured image 200 and the position of the buckle member 243 registered as the reference position 123 is 10 cm or less. The determination unit 137 also determines that the seat position of the seat to be determined is the middle position if the difference in distance between the position of the buckle member 243 detected from the captured image 200 and the position of the buckle member 243 registered as the reference position 123 is 11 cm or more and 20 cm or less. The determination unit 137 also determines that the seat position of the seat to be determined is the front position if the difference in distance between the position of the buckle member 243 detected from the captured image 200 and the position of the buckle member 243 registered as the reference position 123 is 21 cm or more and 30 cm or less.
[0042] Figure 5 is a diagram illustrating a method for determining the seat position based on the position of the waist webbing member 231 in the captured image 200. Figure 5 shows the case where the registered reference position 123 is an intermediate position. When the selection unit 135 selects the waist webbing member 231, the determination unit 137 determines the seat position based on the distance between the position of the waist webbing member 231 detected from the captured image 200 and the previously registered reference position 123. The position of the waist webbing member 231 may be the central position in the height direction of the member detected by the detection unit 133 as the waist webbing member 231 from the captured image 200, or it may be the highest position in the height direction. The height direction refers to the vertical direction in the drawing view of the captured image 200 shown in Figure 5.
[0043] When the difference in distance between the position of the lap webbing member 231 detected from the captured image 200 and the position of the lap webbing member 231 at the reference position 123 is not less than -2 cm and not more than 2 cm, the determination unit 137 determines that the seat position of the seat to be determined is an intermediate position. Further, when the difference in distance between the position of the lap webbing member 231 detected from the captured image 200 and the position of the lap webbing member 231 at the reference position 123 is not less than -6 cm and not more than -3 cm, the determination unit 137 determines that the seat position of the seat to be determined is a forward position. Further, when the difference in distance between the position of the lap webbing member 231 detected from the captured image 200 and the position of the lap webbing member 231 at the reference position 123 is not less than 3 cm and not more than 6 cm, the determination unit 137 determines that the seat position of the seat to be determined is a rear position.
[0044] FIG. 6 is a diagram for explaining a method of determining a seat position based on the position of a shoulder webbing member 233 in a captured image 200. FIG. 6 shows a case where the reference position 123 is a rear position. When the selection unit 135 selects the shoulder webbing member 233, the determination unit 137 determines the seat position based on the deflection amount between the position of the shoulder webbing member 233 detected from the captured image 200 and the pre-registered reference position 123. Specifically, the determination unit 137 compares the trajectory of the member detected as the shoulder webbing member 233 from the captured image 200 by the detection unit 133 with the trajectory of the shoulder webbing member 233 at the reference position 123, and identifies the position where the separation distance in a preset reference direction is maximized. FIG. 6 shows a case where the preset reference direction is the height direction, that is, the vertical direction of the captured image 200 shown in FIG. 6 in a drawing view. The determination unit 137 uses the difference in distance at the position where the identified distance is maximum as the deflection amount, and compares this deflection amount with a threshold value to determine the seat position.
[0045] The determination unit 137 determines that the seat position is at the rear if the maximum difference in distance between the trajectory of the shoulder webbing member 233 detected from the captured image 200 and the trajectory of the shoulder webbing member 233 at the reference position 123 is 0 or greater and less than or equal to the first threshold. The determination unit 137 also determines that the seat position is at the middle if the maximum difference in distance between the trajectory of the shoulder webbing member 233 detected from the captured image 200 and the trajectory of the shoulder webbing member 233 at the reference position 123 is greater than the first threshold and less than or equal to the second threshold. The determination unit 137 also determines that the seat position is at the front if the maximum difference in distance between the trajectory of the shoulder webbing member 233 detected from the captured image 200 and the trajectory of the shoulder webbing member 233 at the reference position 123 is greater than the second threshold and less than or equal to the third threshold. The first, second, and third thresholds satisfy the relationship first threshold < second threshold < third threshold.
[0046] Figure 7 shows a state in which some components of the seat belt device 250 are obscured by an obstacle. In particular, Figure 7 shows a case in which the buckle member 243 is obscured by an obstacle such as a lap blanket and cannot be detected. If the buckle member 243 cannot be detected, the determination unit 137 estimates the position of the buckle member 243 based on the trajectory of at least one of the detectable waist webbing member 231 and shoulder webbing member 233. The determination unit 137 determines the seat position based on the estimated position of the buckle member 243.
[0047] In the example shown in Figure 7, the determination unit 137 determines the trajectories of the members detected as the waist webbing member 231 and the shoulder webbing member 233 from the captured image 200, and estimates the position of the intersection of the extended trajectories as the position of the buckle member 243. The determination unit 137 compares the estimated position of the buckle member 243 with the reference position 123 to determine the seat position.
[0048] When the control device 100 determines the seat position, it controls the operation of the occupant protection device 70 based on the determined seat position when the vehicle 10 collides with an obstacle. For example, the control device 100 controls the timing of airbag deployment, whether or not airbags are deployed, the load on the seat belt device 250, etc., based on the seat position.
[0049] [1-3. Operation of Control Device] FIG. 8 is a flowchart showing the operation of the control device 100. The operation of the control device 100 will be described with reference to the flowchart shown in FIG. 8. First, when the control device 100 is notified from the vehicle ECU 60 that an IG switch has been turned on (step S1), the control device 100 acquires sensor data from the seating detection sensor 40 and a notification signal from the buckle switch 246 (step S2). The control device 100 determines a seat to be determined based on the acquired information (step S3). The control device 100 determines, as a seat to be determined, a seat for which seating of an occupant 20 is detected by the seating detection sensor 40 and wearing of a seat belt is detected by the buckle switch 246.
[0050] Next, the control device 100 determines whether or not there are a plurality of seats determined as determination targets in step S3 (step S4). When there are a plurality of seats determined as determination targets (YES in step S4), the control device 100 acquires a captured image 200 captured by the imaging device 30 from the storage unit 120 (step S5).
[0051] Next, the control device 100 performs image analysis on the acquired captured image 200 to detect a buckle member 243, a lap webbing member 231, and a shoulder webbing member 233 (step S6). Next, the control device 100 determines whether or not at least two members among the buckle member 243, the lap webbing member 231, and the shoulder webbing member 233 have been detected (step S7).
[0052] When at least two members are not detected (NO in step S7), the control device 100 reacquires the captured image 200 (step S5) and performs image analysis on the reacquired captured image 200. When at least two members among the buckle member 243, the lap webbing member 231, and the shoulder webbing member 233 are detected (YES in step S7), the control device 100 selects the detected at least two members as members to be used for determining the seat position (step S8).
[0053] Next, the control device 100 determines the sheet position based on the difference in distance between the position of the member selected in step S8 in the captured image 200 and a preset reference position 123 (step S9). The control device 100 stores the determined sheet position in the storage unit 120 (step S10).
[0054] Next, the control device 100 determines whether or not there are any seats whose seat position has not been determined (step S11). If there are seats whose seat position has not been determined (step S11 / YES), the control device 100 performs image analysis on the captured image 200 to detect the buckle member 243, the waist webbing member 231, and the shoulder webbing member 233 (step S6). If there are no seats whose seat position has not been determined (step S11 / NO), the control device 100 terminates this processing flow.
[0055] Next, we will explain the case where there are not multiple sheets to be judged in step S4. If there are not multiple sheets to be judged (step S4 / NO), the control device 100 acquires the captured image 200 taken by the shooting device 30 from the storage unit 120 (step S12).
[0056] Next, the control device 100 analyzes the acquired captured image to detect the buckle member 243, the waist webbing member 231, and the shoulder webbing member 233 (step S13). Next, the control device 100 determines whether or not it has detected at least one of the buckle member 243, the waist webbing member 231, and the shoulder webbing member 233 (step S14).
[0057] If the control device 100 fails to detect at least one component (step S14 / NO), it reacquires the captured image 200 (step S12). If the control device 100 detects at least one component (step S14 / YES), it selects the at least one component detected by this image analysis as the component to be used to determine the sheet position (step S15).
[0058] Next, the control device 100 determines the sheet position based on the distance between the position of at least one selected member in the captured image 200 and the reference position 123 (step S16). The control device 100 stores the determined sheet position in the storage unit 120 (step S17) and terminates this processing flow.
[0059] [1-4. Modified Version] Figures 9 and 10 are flowcharts showing the operation of the control device 100 in a modified version. In this modified version, the system switches whether or not to continue subsequent processing depending on whether or not a buckle member 243, which provides high seat position determination accuracy, is detected from the captured image 200. If the buckle member 243 is detected by image analysis of the captured image 200, the operation continues; if the buckle member is not detected, the subsequent operation is stopped and the captured image 200 is acquired again.
[0060] Figure 9 shows the operation when the determination in step S4 of Figure 8 determines that there are multiple sheets to be evaluated, and Figure 10 shows the operation when the determination in step S4 of Figure 8 determines that there is only one sheet to be evaluated. Note that the operation in steps S1 to S4 shown in Figure 8 is omitted in Figures 9 and 10.
[0061] The control device 100 acquires the captured image from the storage unit 120 (step T1). Next, the control device 100 analyzes the acquired captured image 200 and determines whether or not the buckle member 243 has been detected (step T2).
[0062] If the control device 100 detects the buckle member 243 from the captured image 200 (step T2 / YES), it sets the value of the variable Fa to "1" (step T3). The variable Fa indicates whether or not the buckle member 243 was detected from the captured image 200. If the control device 100 does not detect the buckle member 243 from the captured image 200 (step T2 / NO), it sets the value of the variable Fa to "0" (step T4), returns to the process of step T1, and reacquires the captured image (step T1). Since the control device 100 could not detect the buckle member 243, it cancels the subsequent processing and returns to the process of step T1.
[0063] When the control device 100 sets the value of variable Fa to "1", it determines whether or not it has detected the shoulder webbing member 233 from the captured image 200 (step T5). If the control device 100 has detected the shoulder webbing member 233 from the captured image 200 (step T5 / YES), it sets the value of variable Fb to "1" (step T6). Variable Fb is a variable that indicates whether or not the shoulder webbing member 233 has been detected from the captured image 200. When the control device 100 sets the value of variable Fb to "1", it proceeds to the process in step T11.
[0064] Furthermore, if the shoulder webbing member 233 is not detected from the captured image 200 (step T5 / NO), the control device 100 sets the value of the variable Fb to "0" (step T7) and proceeds to the process in step T8.
[0065] Next, the control device 100 determines whether or not it has detected the waist webbing member 231 (step T8). If the control device 100 has detected the waist webbing member 231 (step T8 / YES), it sets the value of the variable Fc to "1" (step T9) and proceeds to the process in step T11. The variable Fb is a variable that indicates whether or not the waist webbing member 231 has been detected from the captured image 200. If the waist webbing member 231 has not been detected (step T8 / NO), the control device 100 sets the value of the variable Fc to "0" (step T10) and returns to the process in step T1. Since the control device 100 was unable to detect at least two members, it returns to step T1 to reacquire the captured image 200 and performs image analysis on the captured image again.
[0066] Next, the control device 100 selects the member corresponding to the variable with a value of 1 as the member to be used to determine the seat position (step T11). Specifically, the control device 100 selects the buckle member 243 and the waist webbing member 231 or the shoulder webbing member 233 as the members to be used to determine the seat position. The control device 100 determines the seat position using the selected members (step T12) and stores the seat position determination result in the storage unit 120 (step T13).
[0067] Next, the control device 100 determines whether or not there are any seats whose seat position has not been determined (step T14). If there are seats whose seat position has not been determined (step T14 / YES), the control device 100 returns to step T2 and determines whether or not the buckle member 243 was detected from the captured image 200. If there are no seats whose seat position has not been determined (step T14 / NO), the control device 100 terminates this processing flow.
[0068] Next, the operation when there is only one sheet position to be determined will be explained with reference to the flowchart shown in Figure 10. The control device 100 acquires the captured image from the storage unit 120 (step T15). Next, the control device 100 performs image analysis on the acquired captured image 200 and determines whether or not the buckle member 243 has been detected (step T16).
[0069] If the control device 100 detects the buckle member 243 from the captured image 200 (step T16 / YES), it sets the value of the variable Fa to "1" (step T17). If the control device 100 does not detect the buckle member 243 from the captured image 200 (step T16 / NO), it sets the value of the variable Fa to "0" (step T18), returns to the process of step T1, and reacquires the captured image 200 (step T15).
[0070] Next, when the control device 100 sets the value of variable Fa to "1", it determines whether or not it has detected the shoulder webbing member 233 from the captured image 200 (step T19). If the control device 100 has detected the shoulder webbing member 233 from the captured image 200 (step T19 / YES), it sets the value of variable Fb to "1" (step T20). When the control device 100 sets the value of variable Fb to "1", it proceeds to the process in step T22.
[0071] Furthermore, if the shoulder webbing member 233 is not detected from the captured image 200 (step T19 / NO), the control device 100 sets the value of the variable Fb to "0" (step T21) and proceeds to the process in step T22.
[0072] Next, the control device 100 determines whether or not it has detected the waist webbing member 231 (step T22). If the control device 100 detects the waist webbing member 231 (step T22 / YES), it sets the value of the variable Fc to "1" (step T23) and proceeds to the process in step T25. If the control device 100 does not detect the waist webbing member 231 (step T22 / NO), it sets the value of the variable Fc to "0" (step T24) and proceeds to the process in step T25.
[0073] Next, the control device 100 selects the member corresponding to the variable with a value of "1" as the member to be used to determine the seat position (step T25). Specifically, the control device 100 selects the buckle member 243 and the waist webbing member 231 or the shoulder webbing member 233 as the members to be used to determine the seat position. The control device 100 determines the seat position using the selected members (step T26), stores the seat position determination result in the storage unit 120 (step T27), and ends this processing flow.
[0074] [2. Embodiment 2] Figure 11 is a diagram showing the system configuration of the seat position determination system 1B of Embodiment 2. The seat position determination system 1B shown in Figure 11 is equipped with a wireless communication I / F 80, and the control device 100 is equipped with an acquisition unit 139 as a functional unit. The wireless communication I / F 80 corresponds to the first acquisition unit, and the acquisition unit 139 corresponds to the second acquisition unit.
[0075] The wireless communication interface (I / F) 80 is a communication module that communicates according to NFC (Near Field Communication) communication standards such as Bluetooth and Wi-Fi. Bluetooth and Wi-Fi are registered trademarks. The wireless communication interface (I / F) 80 corresponds to the second acquisition unit. The wireless communication interface (I / F) 80 performs short-range wireless communication with a smart device 300 worn by the crew member 20.
[0076] The determination unit 137 acquires biometric information from the smart device 300 via the wireless communication interface 80. This biometric information includes, for example, the weight and muscle mass of the occupant 20. The determination unit 137 weights the detection results of the buckle member 243, waist webbing member 231, and shoulder webbing member 233 detected by the detection unit 133 based on the biometric information, and determines the seat position based on the weighted detection results of the detection unit 133. For example, if the occupant 20's weight is heavier than a predetermined value, the seat position is determined based on the amount of deflection of the shoulder webbing member 233. If the occupant 20 is large, the seat position may be misjudged as the rear position even if it is in the middle position. Therefore, the determination unit 137 corrects the trajectory of the shoulder webbing member 233 detected by the detection unit 133 from the captured image 200 based on the weight of the occupant 20, and calculates the difference between the corrected trajectory of the shoulder webbing member 233 and the trajectory of the shoulder webbing member 233 registered as the reference position 123.
[0077] For acquiring biological information, the detection unit 133 may perform image analysis on the captured image 200 and estimate the occupant's physique from the analysis results. Alternatively, the occupant's weight may be acquired using the seating detection sensor 40.
[0078] Furthermore, the acquisition unit 139 acquires skeletal information of the crew member 20 based on the analysis results of the captured image 200 from the detection unit 133. For example, the acquisition unit 139 acquires skeletal information based on the positions of joints such as the left and right shoulders and elbows of the upper body of the crew member 20, as well as the position of the head.
[0079] The determination unit 137 determines whether the acquisition unit 139 was able to acquire skeletal information. If the acquisition unit 139 was unable to acquire skeletal information, and the determination in step S7 or S14 above was a negative determination, the determination unit 137 instructs the detection unit 133 to reacquire the captured image 200 and sets an upper limit on the number of times the buckle member 243, waist webbing member 231, and shoulder webbing member 233 are detected from the captured image 200. If the acquisition unit 139 is unable to acquire skeletal information, it is highly likely that the occupant's body is obscured by a blanket or the like, making it impossible to detect the buckle member 243, waist webbing member 231, and shoulder webbing member 233. For this reason, the determination unit 137 sets an upper limit on the number of times the detection unit 133 is instructed to reacquire the captured image 200 if the acquisition unit 139 was unable to acquire skeletal information. Alternatively, the determination unit 137 may set a reacquisition time that allows the detection unit 133 to reacquire the captured image 200 after the determination unit 131 has determined the sheet to be determined.
[0080] [3. Other Embodiments] The embodiments described above are merely one aspect of the present invention and can be arbitrarily modified and applied without departing from the spirit of the present invention. For example, the detection unit 133 detected all of the buckle member 243, waist webbing member 231, and shoulder webbing member 233 from the captured image 200, but a priority order may be set and the members may be detected in the order of the set priority. For example, the priority order can be set so that the buckle member 243 has the highest priority and the shoulder webbing member 233 has the lowest priority. The determination unit 137 determines the seat position based on the members detected by the detection unit 133, but if the determination of the seat position fails, it causes the detection unit 133 to detect the next highest priority member.
[0081] Furthermore, if the seat position cannot be determined after a certain number of attempts, the component used for determining the seat position may be changed. In this case, the posture of the occupant 20, such as resting their elbows on the armrest, may be detected, and the component used for determining the seat position may be changed based on the detection result.
[0082] The functional units of the control device 100 shown in Figures 1 and 11 represent functional configurations, and their specific implementation is not particularly limited. In other words, it is not necessarily required that hardware corresponding to each functional unit be implemented individually, and it is certainly possible to have a configuration in which a single processor executes a program to realize the functions of multiple functional units. Furthermore, some of the functions realized by software in the above embodiment may be realized by hardware, or some of the functions realized by hardware may be realized by software. In addition, the specific detailed configurations of other parts of the control device 100 can also be arbitrarily changed without departing from the spirit of the present invention.
[0083] Furthermore, the processing units in the flowcharts shown in Figures 8, 9, and 10 are divided according to their main processing content in order to facilitate understanding of the processing of the control device 100. The present invention is not limited by the way the processing units are divided or the names of the processing units shown in the flowcharts of Figures 8, 9, and 10. In addition, the processing of the control device 100 can be further divided into more processing units depending on the processing content, or one processing unit can be divided to include even more processing. Also, the processing order in the flowcharts above is not limited to the examples shown.
[0084] Furthermore, while the flowchart in Figure 8 illustrates the case where the sheet position is determined one sheet at a time when there are multiple sheets to be determined, the sheet position of these multiple sheets may be determined simultaneously.
[0085] [Configurations supported by the above embodiment] The above embodiment supports the following configurations.
[0086] (Configuration 1) A seat position determination system comprising: an imaging device for imaging the inside of a moving body; and a control device for determining the seat position of a seat provided on the moving body, wherein the control device includes: a determination unit for determining which seat to be determined from among the seats provided on the moving body; a detection unit for detecting a buckle member, a waist webbing member, and a shoulder webbing member included in a seat belt device from an image captured by the imaging device; a selection unit for selecting at least one member from among the buckle member, the waist webbing member, and the shoulder webbing member to be used for determining the seat position based on the detection result of the detection unit; and a determination unit for determining the seat position based on the detection result of the detection unit for the at least one member selected by the selection unit.
[0087] The seat position detection system of Configuration 1 has a selection unit that selects at least one member to be used to determine the seat position based on the detection results of a buckle member, a waist webbing member, and a shoulder webbing member detected by a detection unit from a captured image. The determination unit then determines the seat position based on the detection result of the detection unit for the at least one member selected by the selection unit. As a result, the seat position can be detected based on the detection results of members that can be detected from a captured image, thus suppressing the influence of the occupant's physique and posture, and improving the accuracy of seat position determination.
[0088] (Configuration 2) The seat position determination system according to Configuration 1, wherein the determination unit determines the seat position based on the distance between a preset reference position of the buckle member and the position of the buckle member detected from the captured image, when the selection of the selection unit includes the buckle member.
[0089] The seat position detection system of configuration 2 determines the seat position based on the distance between a preset reference position of the buckle member and the position of the buckle member detected from the captured image, when the selection unit selects a buckle member. Since the buckle member is less affected by the occupant's physique, determining the seat position based on the position of the buckle member detected from the captured image reduces the influence of the occupant's physique and improves the accuracy of seat position determination.
[0090] (Configuration 3) The seat position determination system according to Configuration 1 or 2, wherein the determination unit determines the seat position based on the distance between a preset reference position of the waist webbing member and the position of the waist webbing member detected from the captured image, when the selection of the selection unit includes the waist webbing member.
[0091] The seat position detection system of configuration 3 determines the seat position based on the distance between a preset reference position of the lumbar webbing member and the position of the lumbar webbing member detected from the captured image, when the selection unit selects the lumbar webbing member. Since the lumbar webbing member is less affected by the occupant's posture, determining the seat position based on the position of the lumbar webbing member detected from the captured image reduces the influence of the occupant's posture and improves the accuracy of seat position determination.
[0092] (Configuration 4) The seat position determination system according to any one of Configurations 1 to 3, wherein the determination unit, when the selection of the selection unit includes the shoulder webbing member, detects a predetermined amount of deflection of the shoulder webbing member relative to a reference line from the captured image, and determines the seat position based on the detected amount of deflection.
[0093] The seat position determination system in configuration 4, when the selection unit selects the shoulder webbing member, detects the amount of deflection of the shoulder webbing member relative to a preset reference line from the captured image and determines the seat position based on the detected amount of deflection. Since the shoulder webbing member is less affected by the occupant's posture, determining the seat position based on the amount of deflection of the shoulder webbing member detected from the captured image reduces the influence of the occupant's posture and improves the accuracy of seat position determination.
[0094] (Configuration 5) The seat position determination system according to any one of Configurations 1 to 4, wherein the determination unit estimates the position of the buckle member based on the trajectory of at least one of the waist webbing member and the shoulder webbing member detected from the captured image, and determines the seat position based on the estimation result of the position of the buckle member.
[0095] The seat position determination system of configuration 5 estimates the position of the buckle member based on at least one trajectory of the waist webbing member and the shoulder webbing member detected from the captured image. Therefore, even if a portion of the waist webbing member and the shoulder webbing member cannot be detected, the seat position can be determined, and a decrease in the accuracy of the seat position determination can be suppressed.
[0096] (Configuration 6) The seat position determination system according to Configuration 5, wherein the determination unit estimates the position of the buckle member at the intersection of the extension lines obtained by extending the trajectories of the waist webbing member and the shoulder webbing member detected from the captured image, and determines the seat position based on the estimated position of the buckle member.
[0097] According to the seat position determination system of configuration 6, the position of the buckle member is estimated by the intersection of the extension lines obtained by extending the trajectories of the waist webbing member and shoulder webbing member detected from the captured image. Therefore, the accuracy of estimating the position of the buckle member can be improved, and a decrease in the accuracy of seat position determination can be suppressed.
[0098] (Configuration 7) The seat position determination system according to any one of Configurations 1 to 6, wherein if the detection unit fails to detect the buckle member, the selection unit does not select any member and causes the detection unit to reacquire the captured image, and then causes the buckle member, the waist webbing member and the shoulder webbing member to re-detect based on the reacquired captured image.
[0099] According to the seat position determination system of configuration 7, if the detection unit cannot detect the buckle member from the captured image, the system does not select any member and instead causes the detection unit to reacquire the image and re-detect the member. Therefore, since the seat position is not determined if the buckle member is not detected, a decrease in the accuracy of seat position determination can be suppressed.
[0100] (Configuration 8) When the determination unit determines a plurality of seats to be determined, the selection unit selects at least two members from the buckle member, the waist webbing member, and the shoulder webbing member, the seat position determination system according to any one of Configurations 1 to 7.
[0101] In the seat position determination system of configuration 8, when multiple seats to be determined are identified, the selection unit selects at least two members from the buckle member, waist webbing member, and shoulder webbing member. When there are multiple seats to be determined, misdetermining may occur if the positions of the members to be determined are close together. Therefore, when multiple seats to be determined are identified, the selection unit selects multiple members. As a result, even when there are multiple seats to be determined, a decrease in the accuracy of seat position determination can be suppressed.
[0102] (Configuration 9) A seat position determination system according to any one of Configurations 1 to 8, comprising a first acquisition unit for acquiring biometric information of an occupant riding in the mobile body, wherein the determination unit weights the detection result of the detection unit for the member selected by the selection unit based on the biometric information, and determines the seat position based on the weighted detection result of the detection unit.
[0103] The seat position determination system of configuration 9 weights the detection result of the detection unit for the member selected by the selection unit based on biological information, and determines the seat position based on the weighted detection result of the detection unit. Therefore, the seat position can be determined by taking into account biological information such as the occupant's physique, and the accuracy of seat position determination can be improved.
[0104] (Configuration 10) A seat position determination system according to any one of items 1 to 9 of Configuration 1, comprising a second acquisition unit for acquiring skeletal information of an occupant riding in the mobile body, wherein the selection unit, if the detection unit cannot detect the buckle member, the waist webbing member, and the shoulder webbing member from the captured image, reacquires the captured image to re-detect the buckle member, the waist webbing member, and the shoulder webbing member, and if the second acquisition unit cannot acquire the skeletal information, limits the number of times the detection unit is instructed to reacquire the captured image.
[0105] The seat position determination system of configuration 10 limits the number of times it reacquires the captured image to detect the buckle member, waist webbing member, and shoulder webbing member if it is unable to obtain the occupant's skeletal information. If the occupant's skeletal information cannot be obtained, the position of the members may not be detectable from the captured image due to the presence of obstacles, and the seat position may not be determined. Therefore, by limiting the number of times the captured image is reacquired when the occupant's skeletal information cannot be obtained, unnecessary processing can be suppressed and the processing load can be reduced.
[0106] (Configuration 11) The seat position determination system according to any one of Configurations 1 to 10, wherein the mobile body is equipped with a occupant protection device to protect the occupants of the mobile body, and the control device controls the occupant protection device based on the seat position when a collision between the mobile body and an obstacle is detected.
[0107] The seat position detection system in configuration 11 controls the occupant protection device based on the seat position when a collision between the moving body and an obstacle is detected. This allows the occupant protection device to be deployed appropriately, ensuring the safety of the occupants.
[0108] (Configuration 12) A control device for determining the seat position of a seat on a moving body, the control device comprising: a determination unit for determining which seat among the seats on the moving body is to be determined for seat position determination; a detection unit for detecting a buckle member, a waist webbing member and a shoulder webbing member included in a seat belt device from an image captured by a camera; a selection unit for selecting at least one member from the buckle member, the waist webbing member and the shoulder webbing member to be used for determining the seat position of the seat, based on the detection result of the detection unit; and a determination unit for determining the seat position based on the detection result of the detection unit of the at least one member selected by the selection unit.
[0109] The seat position determination device of configuration 12 achieves the same effect as the seat position determination system of configuration 1.
[0110] (Configuration 13) A seat position determination method, wherein a processor mounted on a control device for determining the seat position of a seat on a moving body is made to perform the following steps: determining which of the seats on the moving body is to be determined for seat position determination; detecting a buckle member, a waist webbing member, and a shoulder webbing member included in a seat belt device from an image captured by a camera; selecting at least one of the buckle member, the waist webbing member, and the shoulder webbing member to be used for determining the seat position of the seat based on the detection result of the detection step; and determining the seat position of the at least one member selected in the selection step based on the detection result of the detection step.
[0111] The seat position determination method of configuration 13 achieves the same effect as the seat position determination system of configuration 1.
[0112] 1...Seat position determination system, 2...Seat, 10...Vehicle, 20...Occupant, 30...Photography device, 40...Occupancy detection sensor, 50...Touch panel, 60...Vehicle ECU, 70...Occupant protection device, 80...Wireless communication I / F, 100...Control device, 110...Input / output I / F, 120...Storage unit, 121...Control program, 123...Reference position, 130...Processor, 131...Decision unit, 133...Detection unit, 135...Selection unit, 137...Determination unit, 139...Acquisition unit, 200...Captured image, 2 11...Seat cushion, 212...Seat back, 213...Headrest, 220...Seat slide adjuster, 230...Seat belt, 231...Lumbar webbing member, 233...Shoulder webbing member, 240...Retractor, 241...Lower anchor, 243...Buckle member, 245...Tongue plate, 246...Buckle switch, 247...Shoulder anchor, 250...Seat belt device, 270...Center pillar, 300...Smart device.
Claims
1. A seat position determination system comprising: an imaging device for imaging the inside of a moving body; and a control device for determining the seat position of a seat provided on the moving body, wherein the control device includes: a determination unit for determining which seat among the seats provided on the moving body is to be determined for seat position determination; a detection unit for detecting a buckle member, a waist webbing member, and a shoulder webbing member included in a seat belt device from an image captured by the imaging device; a selection unit for selecting at least one member from the buckle member, the waist webbing member, and the shoulder webbing member to be used for determining the seat position based on the detection result of the detection unit; and a determination unit for determining the seat position based on the detection result of the detection unit for the at least one member selected by the selection unit.
2. The seat position determination system according to claim 1, wherein, if the selection of the selection unit includes the buckle member, the determination unit determines the seat position based on the distance between a preset reference position of the buckle member and the position of the buckle member detected from the captured image.
3. The seat position determination system according to claim 1, wherein, if the selection of the selection unit includes the waist webbing member, the determination unit determines the seat position based on the distance between a preset reference position of the waist webbing member and the position of the waist webbing member detected from the captured image.
4. The seat position determination system according to claim 1, wherein, if the selection of the selection unit includes the shoulder webbing member, the determination unit detects a predetermined amount of deflection of the shoulder webbing member relative to a reference line from the captured image, and determines the seat position based on the detected amount of deflection.
5. The seat position determination system according to claim 1, wherein the determination unit estimates the position of the buckle member based on the trajectory of at least one of the waist webbing member and the shoulder webbing member detected from the captured image, and determines the seat position based on the estimation result of the position of the buckle member.
6. The seat position determination system according to claim 5, wherein the determination unit estimates the position of the buckle member at the intersection of the extension lines obtained by extending the trajectories of the waist webbing member and the shoulder webbing member detected from the captured image, and determines the seat position based on the estimated position of the buckle member.
7. The seat position determination system according to claim 1, wherein if the detection unit fails to detect the buckle member, the selection unit does not select any member and causes the detection unit to reacquire the captured image, and then causes the buckle member, the waist webbing member and the shoulder webbing member to re-detect based on the reacquired captured image.
8. When the determination unit determines a plurality of seats to be determined, the selection unit selects at least two members from the buckle member, the waist webbing member, and the shoulder webbing member, the seat position determination system according to claim 1.
9. A seat position determination system according to claim 1, comprising a first acquisition unit for acquiring biometric information of an occupant riding in the mobile body, wherein the determination unit weights the detection result of the detection unit for the member selected by the selection unit based on the biometric information, and determines the seat position based on the weighted detection result of the detection unit.
10. The seat position determination system according to claim 1, further comprising a second acquisition unit for acquiring skeletal information of an occupant riding in the mobile body, wherein the selection unit, if the detection unit cannot detect the buckle member, the waist webbing member, and the shoulder webbing member from the captured image, re-acquires the captured image to re-detect the buckle member, the waist webbing member, and the shoulder webbing member, and if the second acquisition unit cannot acquire the skeletal information, limits the number of times the detection unit is instructed to re-acquire the captured image.
11. The seat position determination system according to claim 1, wherein the mobile body is equipped with an occupant protection device for protecting the occupants of the mobile body, and the control device controls the occupant protection device based on the seat position when a collision between the mobile body and an obstacle is detected.
12. A control device for determining the seat position of a seat on a moving body, the control device comprising: a determination unit for determining which seat among the seats on the moving body is to be determined for seat position determination; a detection unit for detecting a buckle member, a waist webbing member, and a shoulder webbing member included in a seat belt device from an image captured by a camera; a selection unit for selecting at least one member from among the buckle member, the waist webbing member, and the shoulder webbing member to be used for determining the seat position of the seat, based on the detection result of the detection unit; and a determination unit for determining the seat position based on the detection result of the detection unit of the at least one member selected by the selection unit.
13. A method for determining a seat position, wherein a processor mounted on a control device for determining the seat position of a seat on a moving body is made to perform the following steps: determining which seat among the seats on the moving body is to be determined for seat position determination; detecting a buckle member, a waist webbing member, and a shoulder webbing member included in a seat belt device from an image captured by a camera; selecting at least one member from the buckle member, the waist webbing member, and the shoulder webbing member to be used for determining the seat position of the seat, based on the detection result of the detection step; and determining the seat position of the at least one member selected in the selection step based on the detection result of the detection step.