Non-nominal posture detection device and non-nominal posture detection method
The posture collapse detection device improves accuracy by tracking two-dimensional key points on occupant faces, addressing the issue of hidden points in conventional methods, ensuring reliable detection and timely responses.
Patent Information
- Application Number
- PCT/JP2025/016786
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2025-05-08
- Publication Date
- 2026-11-12
AI Technical Summary
Conventional techniques for detecting vehicle occupant posture collapse based on connection angles of attention points are unreliable when one of the predetermined attention points is hidden, leading to inaccurate or missed detections.
A posture collapse detection device that tracks two-dimensional key points on occupant faces using a tracking unit, calculating their movement to detect posture collapse, even when one point is obscured, by employing a tracking unit that utilizes template matching to maintain tracking.
Enhances the accuracy of posture collapse detection by ensuring continuous tracking and detection even when partial face obscuration occurs, providing timely warnings or control actions.
Smart Images

Figure JP2025016786_12112026_PF_FP_ABST
Abstract
Description
Posture collapse detection device and posture collapse detection method
[0001] The present disclosure relates to a posture collapse detection device and a posture collapse detection method.
[0002] Conventionally, a technique is known for detecting the posture collapse of an occupant based on a captured image obtained by imaging the occupant in a vehicle cabin. For example, Patent Document 1 discloses a technique of calculating a connection angle of attention points, which is the inclination of a connection of attention points formed by a line segment connecting two attention points selected from attention points including joint points of the upper body and face part points of a vehicle driver, and estimating the lateral falling posture of the driver from the connection angle of attention points.
[0003] International Publication No. 2021 / 166791
[0004] The conventional technique for detecting the posture collapse of a driver from the connection angle of attention points as disclosed in Patent Document 1 is premised on that both of two predetermined attention points are detected. However, for example, a part of the upper body or face of the occupant may be hidden by a shielding object, so that a part of the upper body or face of the occupant may not be temporarily detected on the captured image. If an attempt is made to detect the posture collapse of the occupant using the conventional technique as described above, when even any one of the two predetermined attention points is not detected, the connection angle of attention points cannot be calculated, and there is a problem that the posture collapse of the occupant may not be detected or may be erroneously detected.
[0005] The present disclosure has been made to solve the above-described problems, and an object thereof is to provide a posture collapse detection device capable of detecting the posture collapse of a vehicle occupant with higher accuracy than the conventional technique that detects posture collapse on the premise that two predetermined attention points are detected.
[0006] The posture collapse detection device according to this disclosure comprises: an image acquisition unit that acquires captured images in a time series in which at least the face of an occupant in the vehicle interior is captured; a key point detection unit that detects two-dimensional key points corresponding to key points, which are points indicating the joints or parts of the occupant's face, on the captured images based on the captured images acquired by the image acquisition unit; a tracking unit that tracks the two-dimensional key point positions, which are the positions of the two-dimensional key points detected by the key point detection unit on the captured images, in a time series and calculates the amount of movement of the two-dimensional key points in the captured images; and a posture collapse detection unit that detects posture collapse of the occupant based on the amount of movement of the two-dimensional key points in the captured images calculated by the tracking unit.
[0007] According to this disclosure, the posture collapse detection device can detect posture collapse of vehicle occupants with greater accuracy than conventional technologies that rely on the detection of two predetermined points of interest.
[0008] This figure shows an example of the configuration of the posture collapse detection device according to Embodiment 1. This figure is for explaining an example of a method for tracking the position of a two-dimensional key point by the tracking unit when a two-dimensional key point is not detected by the key point detection unit in Embodiment 1. Figure 2A shows an example of an image captured of an occupant seated in a normal posture inside a vehicle. Figure 2B shows the occupant's face region and face orientation detected based on the image captured in Figure 2A. Figure 2C shows the occupant's face region and the two-dimensional key point detected based on the occupant's three-dimensional head position and face orientation. Figure 2D is for explaining the motion vector calculated by tracking the position of the two-dimensional key point. The following are other figures illustrating an example of a method for tracking the position of a two-dimensional keypoint by the tracking unit when a two-dimensional keypoint is not detected by the keypoint detection unit in Embodiment 1. Figure 3A shows an example of a small-region template set with the position of a two-dimensional keypoint on the captured image as the reference position. Figure 3B shows an example of a search area set on the captured image. Figure 3C shows the pixel that the tracking unit determined to have the best match with the pixel and pattern included in the corresponding small-region template in each search area shown in Figure 3B. Figures 4A, 4B, and 4C are figures illustrating the principle of motion vector detection by template matching of the small-region template by the tracking unit in Embodiment 1. This diagram illustrates the concept of detecting occupant posture collapse in Embodiment 1, based on an average vector showing the average of the occupant's face region, face orientation, and the amount of movement of two-dimensional keypoints in the captured image, by the posture collapse detection unit. Figure 5A shows an example of a captured image of an occupant in a normal posture state, and Figure 5B shows an example of a captured image when the occupant is experiencing posture collapse, such as leaning against the door with their head or shoulders tilted laterally toward the door.The following are additional diagrams illustrating the concept of detecting occupant posture collapse in Embodiment 1, based on an average vector showing the average of the movement amounts of the occupant's face region, face orientation, and two-dimensional keypoints in the captured image, by the posture collapse detection unit, where Figure 6A is an example of a captured image in which the occupant is in a normal posture state, and Figure 6B is an example of a captured image in which the occupant is in a posture collapse state where they are leaning backward. The following are additional diagrams illustrating the concept of detecting occupant posture collapse in Embodiment 1, based on an average vector showing the average of the movement amounts of the occupant's face region, face orientation, and two-dimensional keypoints in the captured image, where Figure 7A is an example of a captured image in which the occupant is in a normal posture state, and Figure 7B is an example of a captured image in which the occupant is in a posture collapse state where they are leaning forward. The following are other diagrams illustrating the concept of detecting occupant posture collapse in Embodiment 1, based on an average vector showing the average of the movement of the occupant's face region, face orientation, and two-dimensional keypoints in the captured image, by the posture collapse detection unit. Figure 8A shows an example of a captured image of an occupant in a normal posture state, and Figure 8B shows an example of a captured image when the occupant is leaning against a vehicle structure diagonally in front, and their entire body is tilted diagonally forward, causing the occupant to move out of the field of view of the imaging device. The following are diagrams illustrating an example of the tracking unit failing to track in Embodiment 1. Figure 9A shows an example of a captured image taken at time T, and Figure 9B shows an example of a captured image taken at time (T+ΔT). The following are other diagrams illustrating another example of the tracking unit failing to track in Embodiment 1. Figure 10A shows an example of a captured image taken at time T, and Figure 10B shows another example of a captured image taken at time (T+ΔT). This is a flowchart illustrating an example of operation of the posture collapse detection device according to Embodiment 1. Figures 12A and 12B show an example of the hardware configuration of the posture collapse detection device according to Embodiment 1.
[0009] The embodiments of this disclosure will now be described in detail with reference to the drawings. Embodiment 1. The posture collapse detection device according to Embodiment 1 detects posture collapse of an occupant based on an image in which at least the face of the occupant present in the vehicle cabin is captured. In Embodiment 1 below, the occupant of the vehicle whose posture collapse is detected by the posture collapse detection device is assumed to be the driver, for example. However, this is only an example, and the occupant of the vehicle whose posture collapse is detected by the posture collapse detection device may be, for example, an occupant sitting in the passenger seat or an occupant sitting in the rear seat.
[0010] Figure 1 shows an example of the configuration of the posture collapse detection device 1 according to Embodiment 1. The posture collapse detection device 1 according to Embodiment 1 is assumed to be mounted on a vehicle (not shown). The posture collapse detection device 1 is connected to an imaging device 2 via a network. The imaging device 2 is mounted on the vehicle. The imaging device 2 may be a visible light camera or an infrared camera. If the imaging device 2 is an infrared camera, the infrared camera is provided with a light source (not shown) that irradiates infrared light for imaging to an area including the occupant's face. This light source is composed of, for example, an LED (Light Emitting Diode). The imaging device 2 is installed, for example, on the steering column, pillar, or meter panel of the vehicle, so as to be able to image occupants that may be present in the vehicle interior from the front. The imaging device 2 is installed so as to be able to image at least the occupant's face. The imaging device 2 does not need to be a wide-angle camera intended to capture the entire body of the occupant; for example, it may be a narrow-angle camera capable of capturing the occupant's head and shoulders when the occupant is in a normal posture inside the vehicle. The imaging device 2 may also be shared with a so-called DMS (Driver Monitoring System). In Embodiment 1, an occupant in a normal posture means that the occupant is seated with their buttocks in deep contact with the seat surface, pressed against the backrest, and their head in an appropriate position (for example, at the height of the headrest). The imaging device 2 outputs the captured image (hereinafter referred to as "captured image") to the posture collapse detection device 1.
[0011] The posture collapse detection device 1 detects posture collapse of occupants present in the vehicle based on the images captured by the imaging device 2. More specifically, the posture collapse detection device 1 detects whether or not an occupant present in the vehicle is experiencing posture collapse based on the images captured by the imaging device 2. In Embodiment 1, posture collapse refers to a state in which the body is unstable, deviating from a normal posture. Posture collapse includes, for example, "leaning," where the head or trunk is biased to the side, forward, or backward; "leaning forward," where the upper body is excessively tilted forward; or "leaning backward," where the waist or back is excessively arched. In Embodiment 1, for example, a posture of looking widely to the side may also be considered posture collapse. If an occupant is experiencing posture collapse, it is assumed that, for example, if the occupant is the driver, it may impair their ability to drive, or that they may require emergency attention due to poor health. When the posture collapse detection device 1 detects that an occupant is experiencing posture collapse, it outputs a warning from, for example, an output device (not shown). The output device is, for example, a speaker mounted on the vehicle. Furthermore, if the vehicle is capable of autonomous driving, for example, when the posture collapse detection device 1 detects that an occupant is experiencing posture collapse, it may instruct the autonomous driving control device (not shown) to take control such as stopping the vehicle or pulling over to the side of the road.
[0012] The posture collapse detection device 1 comprises an image acquisition unit 11, a calibration unit 12, a face region detection unit 13, a three-dimensional information detection unit 14, a key point detection unit 15, a tracking unit 16, a posture collapse detection unit 17, and a storage unit 18.
[0013] The image acquisition unit 11 acquires captured images from the imaging device 2. The image acquisition unit 11 acquires captured images frame by frame in chronological order. The image acquisition unit 11 outputs the acquired captured images to the calibration unit 12, the face region detection unit 13, or the key point detection unit 15. The image acquisition unit 11 also stores the acquired captured images in chronological order in the storage unit 18, associating them with the date and time the images were acquired.
[0014] The calibration unit 12 performs calibration of a three-dimensional model of a person's head, used to detect the occupant's head position in real space (hereinafter referred to as "three-dimensional head position") or face orientation, based on the captured images acquired by the image acquisition unit 11 during a preset period (hereinafter referred to as the "calibration period"). The calibration period is a period such as 30 seconds or 1 minute, and is set in advance by the developer, etc. The developer, etc. sets the calibration period to the period during which it is assumed that the occupant is in a normal posture after getting into the vehicle. Generally, it is assumed that a person is seated in a normal posture for a while immediately after getting into the vehicle. The three-dimensional model of the person's head is, for example, a three-dimensional model that reflects the size of a standard adult face and the feature points of the facial features. In the following embodiment 1, the three-dimensional model of the person's head is also simply referred to as the "three-dimensional model". The three-dimensional model is prepared in advance and stored in a location accessible to the posture collapse detection device 1, such as the memory unit 18. In the three-dimensional model, the feature points indicating the facial features are known in advance. Facial features include the outer corners of the eyes, inner corners of the eyes, the tip of the nose, or the corners of the mouth. In the following Embodiment 1, feature points indicating facial features on the 3D model are also called "model feature points." The detection of the occupant's 3D head position or face orientation using the 3D model is performed by the 3D information detection unit 14. Details of the 3D information detection unit 14 will be described later.
[0015] The calibration unit 12 can perform calibration of the 3D model using known calibration techniques that calibrate the 3D model based on captured images of a person's face, so a detailed explanation will be omitted. However, for example, the calibration unit 12 performs calibration of the 3D model as follows. For example, the calibration unit 12 extracts feature points that indicate parts of the occupant's face, such as the outer corners of the eyes, inner corners of the eyes, tip of the nose, or corners of the mouth, based on the captured images acquired by the image acquisition unit 11. The calibration unit 12 extracts multiple feature points. The calibration unit 12 may extract feature points using an appropriate method. For example, the calibration unit 12 may extract feature points using known machine learning techniques. Alternatively, for example, the calibration unit 12 may extract feature points using known image recognition techniques such as edge extraction or template matching. Next, the calibration unit 12 places the pre-prepared 3D model in a virtual 3D space represented in the camera coordinate system (hereinafter referred to as "virtual 3D space") at a predetermined reference position and in a predetermined reference orientation. The reference position and reference orientation are the positions and orientations that would be expected when a standard adult is facing forward in a normal posture, respectively. Subsequently, the calibration unit 12 compares the positional relationship between the feature points extracted from the captured image and the corresponding model feature points on the 3D model, and adjusts the model shape of the 3D model so that the difference is minimized. The calibration unit 12 repeats this adjustment for the calibration period. By performing the calibration process as described above, the calibration unit 12 makes the 3D model closer to the shape of the occupant's face. The calibration unit 12 can adjust a pre-prepared 3D model so that it reflects differences in face shape due to race or individual differences, such as nose height. When the calibration process is complete, the calibration unit 12 outputs information indicating that the calibration process is complete (hereinafter referred to as "calibration completion information") to the face region detection unit 13.
[0016] The face region detection unit 13 detects the region in which the occupant's face is captured (hereinafter referred to as the "face region") based on the captured image acquired by the image acquisition unit 11. The face region detection unit 13 may detect the occupant's face region using known techniques such as pattern matching, image recognition processing techniques, or techniques using machine learning-based face detection algorithms. The face region is, for example, a rectangular region in the captured image that is determined to contain the occupant's face, and is a region that encompasses the contour of the face or parts of the face. The face region detection unit 13 outputs information indicating the detected face region (hereinafter referred to as "face region information") to the 3D information detection unit 14. The face region information is, for example, information to which the occupant's face region can be identified by adding information to the captured image. If the face region detection unit 13 is unable to detect the face region based on the captured image, it outputs face region information to the 3D information detection unit 14 indicating that the face region could not be detected, i.e., undefined.
[0017] The 3D information detection unit 14 detects the occupant's 3D head position and face orientation based on the face region detected by the face region detection unit 13 and the 3D model. In Embodiment 1, the occupant's 3D head position is defined, for example, by the center of the 3D positions of the inner corners of both of the occupant's eyes. The 3D head position is expressed, for example, in a real-space coordinate system (world coordinate system). The face orientation is expressed by the yaw angle, pitch angle, and roll angle.
[0018] An example of a method for detecting the three-dimensional head position or face orientation of an occupant by the three-dimensional information detection unit 14 in Embodiment 1 will be described. First, the three-dimensional information detection unit 14 extracts feature points indicating parts of the occupant's face, such as the outer corners of the eyes, inner corners of the eyes, tip of the nose, or corners of the mouth, from the face region detected by the face region detection unit 13. The three-dimensional information detection unit 14 may extract feature points by an appropriate method. For example, the three-dimensional information detection unit 14 may extract feature points using known machine learning techniques, or it may extract feature points using known image recognition techniques such as edge extraction or template matching. Note that the extraction of parts of the occupant's face may be performed together with the detection of the face region by the face region detection unit 13 when it detects the face region. Next, the three-dimensional information detection unit 14 places the three-dimensional model in a virtual three-dimensional space, for example, at a reference position and in a reference orientation. Note that the three-dimensional model used here is a three-dimensional model that has been calibrated by the calibration unit 12. The 3D information detection unit 14 compares feature points extracted from the face region with multiple model feature points on the 3D model corresponding to those feature points on the captured image, and rotates or translates the 3D model so that the error between the positions of the multiple feature points and the positions of the multiple model feature points on the captured image is within a preset threshold (hereinafter referred to as the "error determination threshold"). In this process, for example, if the 2D coordinates of the feature points on the 2D image and the 3D coordinates of the corresponding feature points on the 3D image are known, the 3D information detection unit 14 can calculate the coordinates of the model feature points on the captured image using the DLT (Direct Linear Transform) method, which derives a perspective projection matrix that projects the 3D coordinates onto the 2D image. The 3D information detection unit 14 then adjusts the position and orientation of the model so that the error is minimized, and detects the occupant's 3D head position and face orientation from the rotation matrix or translation matrix obtained from the result.
[0019] The 3D information detection unit 14 outputs information indicating the detected 3D head position and face orientation of the occupant (hereinafter referred to as "3D detection information") to the key point detection unit 15 and the posture collapse detection unit 17. The 3D information detection unit 14 also stores the 3D detection information in the storage unit 18, associating it with the captured image that was used to detect the occupant's 3D head position and face orientation. If the face region detection unit 13 fails to detect a face region, that is, if the face region detection unit 13 outputs face region information indicating that the face region is undefined, the 3D information detection unit 14 determines that the occupant's 3D head position and face orientation could not be detected, i.e., they are undefined, and outputs 3D detection information indicating that the 3D head position and face orientation are undefined to the key point detection unit 15 and the posture collapse detection unit 17, and also stores it in the storage unit 18.
[0020] The key point detection unit 15 detects points corresponding to the occupant's key points (hereinafter referred to as "2D key points") on the captured image based on the image acquired by the image acquisition unit 11. In Embodiment 1, a key point is a point indicating a joint or part of the face of a person. Examples of key points include points indicating the top of the head, neck, right shoulder, or left shoulder. The developers have predetermined which points among the points indicating the occupant's joint or face to be used as key points for detection by the key point detection unit 15. The developers have predetermined appropriate points indicating at least the joints or parts of the face above both shoulders as key points. It is sufficient that at least one type of point indicating a joint or part of the face, in other words, one point indicating a joint or part of the face, be set as a key point, but in Embodiment 1 below, as an example, four points—the top of the head, neck, right shoulder, and left shoulder—are set as key points. In other words, the key point detection unit 15 detects two-dimensional key points corresponding to the occupant's head, neck, right shoulder, and left shoulder on the captured image, based on the image acquired by the image acquisition unit 11. The positions of the two-dimensional key points are represented by coordinates on the captured image.
[0021] The details of the detection of two-dimensional keypoints by the keypoint detection unit 15 will be explained with an example. For example, the keypoint detection unit 15 detects two-dimensional keypoints based on the occupant's face region detected by the face region detection unit 13 and the occupant's three-dimensional head position and face orientation detected by the three-dimensional information detection unit 14.
[0022] For example, developers collect data representing the shape or structure of a person's upper body beforehand. For instance, developers collect models (hereinafter referred to as "human body models") that reproduce the upper body of a person in three-dimensional space with various facial orientations. These human body models could be, for example, models obtained by 3D scanning an actual person, or average shape models generated by statistically processing a large amount of human body data. Based on the collected data, developers statistically calculate the relative positional relationship between the three-dimensional head position and the three-dimensional positions of key points (here, the top of the head, neck, right shoulder, and left shoulder) according to the facial orientation. This allows developers to generate rules (hereinafter referred to as "3D key point estimation rules") for predicting the three-dimensional positions of key points from the three-dimensional head position according to the facial orientation. The 3D key point estimation rules define the relative positional relationship between the three-dimensional head position and the three-dimensional positions of key points according to the facial orientation.
[0023] The keypoint detection unit 15 detects 2D keypoints, for example, in accordance with the 3D keypoint estimation rules that have been generated in advance as described above. Specifically, the keypoint detection unit 15 calculates the scaling ratio of how much the size (vertical and horizontal) of the occupant's face region on the captured image is reduced or enlarged compared to the size of the face on the captured image of a reference person. The size of the face on the captured image of the reference person is set in advance by the developers, etc. For example, the developers estimate how large the face of a person with a standard build would appear on the captured image if that person were seated in a standard position in a normal posture, and set the size of the face on the captured image of the reference person based on this. The keypoint detection unit 15 only needs to acquire the face region information output from the face region detection unit 13 via the 3D information detection unit 14.
[0024] The keypoint detection unit 15 then estimates the three-dimensional position of the keypoint based on the three-dimensional keypoint estimation rule, and detects the two-dimensional keypoint based on the estimated three-dimensional position of the keypoint and the above-mentioned scaling ratio. More specifically, the keypoint detection unit 15 calculates the relative distance from the occupant's three-dimensional head position to the three-dimensional position of the keypoint, corresponding to the occupant's face orientation, according to the three-dimensional keypoint estimation rule, based on the occupant's three-dimensional head position and face orientation detected by the three-dimensional information detection unit 14. In other words, it calculates the relative positional relationship between the occupant's three-dimensional head position and the three-dimensional position of the keypoint. Based on this relative positional relationship, the keypoint detection unit 15 uses known perspective projection techniques to project a point indicating the occupant's three-dimensional head position and a point indicating the three-dimensional position of the keypoint onto the captured image, and tentatively determines the position of the keypoint on the captured image. As a result of the projection, the keypoint detection unit 15 can tentatively determine where the three-dimensional position of the occupant's keypoint in three-dimensional space corresponds on the captured image. The keypoint detection unit 15 adjusts the position of the tentatively determined keypoints on the captured image by applying a scaling ratio determined from the size (vertical and horizontal) of the occupant's face region on the captured image. The keypoint detection unit 15 detects the adjusted keypoints on the captured image as two-dimensional keypoints. In this way, the keypoint detection unit 15 only needs to detect two-dimensional keypoints.
[0025] In the example above, the 3D keypoint estimation rule was defined as a rule that defines the relative positional relationship between the person's 3D head position and the 3D position of the keypoint, depending on the orientation of the person's face. However, this is only one example. For example, the 3D keypoint estimation rule could be a rule that defines the relative positional relationship between the person's 3D head position and the 3D position of the keypoint, without considering the orientation of the person's face. In this case, the keypoint detection unit 15 can calculate the relative distance from the occupant's 3D head position to the 3D position of the keypoint, in other words, the relative positional relationship between the occupant's 3D head position and the 3D position of the keypoint, based on the 3D head position of the occupant detected by the 3D information detection unit 14, according to the 3D keypoint estimation rule, to tentatively determine the position of the keypoint on the captured image. Then, by applying the scaling factor obtained from the size (vertical and horizontal) of the occupant's face region on the captured image, it can detect the 2D keypoint.
[0026] Furthermore, if there are multiple keypoints, the keypoint detection unit 15 estimates the positional relationship between the occupant's 3D head position and the 3D position of each keypoint according to the 3D keypoint estimation rules, projects this positional relationship onto the captured image with a scaling factor applied, and detects the 2D keypoint of each keypoint.
[0027] When the keypoint detection unit 15 detects a two-dimensional keypoint, it outputs information indicating the detected two-dimensional keypoint (hereinafter referred to as "two-dimensional keypoint information") to the tracking unit 16. The two-dimensional keypoint information is, for example, information that associates information indicating which keypoint it is with information indicating the location of the two-dimensional keypoint. The information indicating which keypoint it is is, for example, information that can identify it as the top of the head, neck, right shoulder, or left shoulder. The information indicating the location of the two-dimensional keypoint is the coordinate on the captured image that indicates the location of the two-dimensional keypoint. If a two-dimensional keypoint cannot be detected, the information indicating the location of the two-dimensional keypoint will be, for example, information indicating that it is undetermined. The keypoint detection unit 15 also stores the two-dimensional keypoint information in the storage unit 18, associating it with the captured image that was used to detect the occupant's three-dimensional head position and face orientation.
[0028] The tracking unit 16 tracks the position of the two-dimensional keypoint detected by the keypoint detection unit 15 on the captured image (hereinafter referred to as "two-dimensional keypoint position") in a time series. The tracking unit 16 then calculates the amount of movement of the two-dimensional keypoint in the captured image. More specifically, this amount of movement is assumed to be the amount of movement from, for example, the position of the two-dimensional keypoint in the captured image one frame ago to the position of the two-dimensional keypoint in the current captured image. This amount of movement is represented by a motion vector. The tracking unit 16 also calculates the average of the amount of movement of each two-dimensional keypoint in the captured image. In Embodiment 1, the average of the motion vectors representing the amount of movement of each two-dimensional keypoint in the captured image is also called the average vector. That is, the average of the amount of movement of each two-dimensional keypoint in the captured image is represented by the average vector. For example, the tracking unit 16 can track the time series of 2D keypoint positions by detecting the difference between the 2D keypoint positions detected by the keypoint detection unit 15 for the image captured one frame ago and the 2D keypoint positions detected by the keypoint detection unit 15 for the current image. In this case, the motion vector can be defined, for example, as the difference between the coordinates of the 2D keypoint position one frame ago and the corresponding coordinates of the 2D keypoint position in the current frame.
[0029] However, there may be cases where the keypoint detection unit 15 cannot detect a two-dimensional keypoint, or in other words, detects it as indeterminate, for reasons such as the occupant's face being obscured by an obstruction, or the occupant experiencing a significant change in posture. In Embodiment 1, the tracking unit 16 can continue tracking the two-dimensional keypoint position without interrupting the tracking of the two-dimensional keypoint position, even if the keypoint detection unit 15 is unable to detect a two-dimensional keypoint in the current captured image.
[0030] An example of a method for tracking the position of a two-dimensional keypoint by the tracking unit 16 when the keypoint detection unit 15 does not detect a two-dimensional keypoint will be described. If the keypoint detection unit 15 does not detect a two-dimensional keypoint, for example, the tracking unit 16 sets a small region template that includes a pixel corresponding to the reference position (hereinafter referred to as "reference pixel"), using the two-dimensional keypoint position of a two-dimensional keypoint detected by the keypoint detection unit 15 based on past captured images as the reference position. On the captured image acquired by the image acquisition unit 11, in other words, on the current captured image, it sets a search area wider than the small region template by extending it by a number of pixels (hereinafter referred to as "search area setting pixels") determined in the vertical and horizontal directions centered on the reference pixel, and tracks the position of the two-dimensional keypoint by comparing the pixels included in the search area with the pixels included in the small region template. The two-dimensional keypoint position of a two-dimensional keypoint detected based on past captured images may be, for example, the most recent two-dimensional keypoint position among the detected two-dimensional keypoint positions. The tracking unit 16 can identify the latest 2D keypoint position by referring to the storage unit 18. In this case, the tracking unit 16 sets, for example, the region of the reference pixel as a small region template.
[0031] Here, an example of a method for tracking the position of a two-dimensional key point by the tracking unit 16 when a two-dimensional key point is not detected by the key point detection unit 15, as described above, will be explained in more detail with reference to the drawings. Figures 2, 3, and 4 are diagrams illustrating an example of a method for tracking the position of a two-dimensional key point by the tracking unit 16 when a two-dimensional key point is not detected by the key point detection unit 15 in Embodiment 1. Figure 2A is a diagram showing an example of an image taken of an occupant seated in a normal posture in the interior of a vehicle at time T, and Figure 2B is a diagram showing the face region and face orientation of the occupant detected based on the image taken at time T shown in Figure 2A. In Figure 2A, the occupant is indicated by "009". Note that the reference numerals for the occupant are omitted in Figure 2B. In Figure 2B, the face region of the occupant is indicated by "010", and the face orientation of the occupant is indicated by a black arrow. As described above, the face region detection unit 13 detects the occupant's face region based on the captured image acquired by the image acquisition unit 11, and the 3D information detection unit 14 detects the occupant's 3D head position and face orientation based on the face region detected by the face region detection unit 13 and the 3D model. For convenience, the occupant's face orientation is shown, but the occupant's 3D head position is not shown. Figure 2C shows the occupant's face region and the 2D keypoints detected based on the occupant's 3D head position and face orientation. In Figure 2C, the top of the head as a 2D keypoint is indicated by "011", the neck as a 2D keypoint is indicated by "012", the right shoulder as a 2D keypoint is indicated by "013", and the left shoulder as a 2D keypoint is indicated by "014". As described above, the keypoint detection unit 15 detects 2D keypoints on the captured image. Figure 2D will be described later.
[0032] Let's assume that the time is now (T + ΔT), where "ΔT" is the frame interval. Also, let's assume that the keypoint detection unit 15 failed to detect any 2D keypoints. For convenience, we will assume that the keypoint detection unit 15 failed to detect any 2D keypoints among the keypoints on the top of the head, neck, right shoulder, and left shoulder. Figure 3 illustrates an example of the specific procedure for tracking the 2D keypoint position by the tracking unit 16 when no 2D keypoints are detected.
[0033] If no 2D keypoint is detected, the tracking unit 16 first sets a small region template including a reference pixel, using the 2D keypoint position of a 2D keypoint already detected by the keypoint detection unit 15 based on past captured images as the reference position. Here, assume that the most recent 2D keypoint position detected by the keypoint detection unit 15 was the 2D keypoint position of a 2D keypoint (see Figure 2C) detected on the captured image taken at time T. In this case, the tracking unit 16 sets a small region template including a reference pixel corresponding to the 2D keypoint, using the 2D keypoint position of a 2D keypoint already detected based on the captured image taken at time T as the reference position. Figure 3A shows an example of a small region template set with the 2D keypoint position on the captured image taken at time T as the reference position. In Figure 3A, the small region template (hereinafter referred to as the "first small region template") set with the 2D keypoint position at the top of the head as the reference position is indicated by "111". Furthermore, in Figure 3A, the small region template (hereinafter referred to as the "second small region template") set up with the 2D keypoint position of the neck as the reference position is indicated by "112". Also in Figure 3A, the small region template (hereinafter referred to as the "third small region template") set up with the 2D keypoint position of the right shoulder as the reference position is indicated by "113". Furthermore, in Figure 3A, the small region template (hereinafter referred to as the "fourth small region template") set up with the 2D keypoint position of the left shoulder as the reference position is indicated by "114".
[0034] Next, the tracking unit 16 sets a search area on the captured image taken at time (T + ΔT), extending vertically and horizontally by the number of pixels used for setting the search area, centered on the reference pixel. Figure 3B shows an example of a search area set on the captured image taken at time (T + ΔT). In Figure 3B, the search area set centered on the reference pixel corresponding to the 2D keypoint position (reference position) of the top of the head (referred to as the "first search area") is indicated by "211". The first search area is larger than the first sub-region template. Also in Figure 3B, the search area set centered on the reference pixel corresponding to the 2D keypoint position (reference position) of the neck (referred to as the "second search area") is indicated by "212". The second search area is larger than the second sub-region template. Also in Figure 3B, the search area set centered on the reference pixel corresponding to the 2D keypoint position (reference position) of the right shoulder (referred to as the "third search area") is indicated by "213". The third search area is larger than the third sub-region template. Also, in Figure 3B, the search area (referred to as the "fourth search area") set centered on the reference pixel corresponding to the 2D keypoint position (reference position) in the upper left corner is indicated by "214". The fourth search area is larger than the fourth sub-region template.
[0035] The tracking unit 16 then tracks the two-dimensional keypoint position by comparing the pixels included in the search area with the pixels included in the sub-region template. Specifically, the tracking unit 16 tracks the position of the pixel that best matches the pattern of the pixels included in the sub-region template among the pixels included in the search area, as the two-dimensional keypoint position. Here, "matching the pattern of the pixels included in the sub-region template" means that the two-dimensional distribution (pattern or texture) of the pixel values included in the sub-region template matches. Figure 3C is a diagram showing the pixels that the tracking unit 16 determined to have the best match with the pattern of the pixels included in the corresponding sub-region template in each search area shown in Figure 3B. In Figure 3C, the pixel that was determined to have the best match with the pixels included in the first sub-region template in the first search area is indicated by "311". Also in Figure 3C, the pixel that was determined to have the best match with the pixels included in the second sub-region template in the second search area is indicated by "312". Furthermore, in Figure 3C, the pixel that was determined to be the best match with the pixels included in the third sub-region template in the third search region is indicated as "313". Also in Figure 3C, the pixel that was determined to be the best match with the pixels included in the fourth sub-region template in the fourth search region is indicated as "314".
[0036] For example, if the image captured at time T is the image captured one frame before the image captured at time (T + ΔT), then the amount of movement from the pixel of the first sub-region template indicated by "111" in Figure 3A to the pixel indicated by "311" in Figure 3B is the motion vector (referred to as the "first motion vector"). Also, the amount of movement from the pixel of the second sub-region template indicated by "112" in Figure 3A to the pixel indicated by "312" in Figure 3B is the motion vector (referred to as the "second motion vector"). Furthermore, the amount of movement from the pixel of the third sub-region template indicated by "113" in Figure 3A to the pixel indicated by "313" in Figure 3B is the motion vector (referred to as the "third motion vector"). Furthermore, the amount of movement from the pixel of the fourth sub-region template indicated by "114" in Figure 3A to the pixel indicated by "314" in Figure 3B is the motion vector (referred to as the "fourth motion vector").
[0037] Figure 2D is a diagram illustrating the first motion vector, second motion vector, third motion vector, and the average vector of these motion vectors. In Figure 2D, the first motion vector is indicated by "015", the second motion vector by "016", the third motion vector by "017", and the fourth motion vector by "018". Also in Figure 2D, the first motion vector, second motion vector, third motion vector, and the average vector of the third motion vector are indicated by the white arrow "019".
[0038] The principle of motion vector detection by template matching of a small-region template is described below. Figures 4A, 4B, and 4C are diagrams illustrating the principle of motion vector detection by template matching of a small-region template by the tracking unit 16 in Embodiment 1. Figure 4A is a diagram showing an example of a small-region template that includes a reference pixel corresponding to a two-dimensional keypoint position detected on an image captured at time T. In Figure 4A, the small-region template is indicated by "115". The coordinates of the two-dimensional keypoint position are (px, py), and the small-region template is width w × height h. The pixel value within the small-region template is M. T Let it be represented as (i, j). Note that (px-w / 2 ≤ i ≤ px+w / 2, py-h / 2 ≤ j ≤ py+h / 2).
[0039] Figure 4B shows an example of a search area set on an image captured at time (T + ΔT), where the search area is expanded vertically and horizontally by the number of pixels used for setting the search area, centered on the reference pixel. In Figure 4B, the search area is indicated by "215". The search area is defined as horizontal rx × vertical ry. The pixel value within the search area is R. T+ΔT Let it be represented as (i, j). Note that (px - rx / 2 ≤ i ≤ px + rx / 2, py - ry / 2 ≤ j ≤ py + ry / 2).
[0040] One example of template matching is a method that minimizes the sum of differences. For example, when a template on an image captured at time T is shifted by horizontal dx and vertical dy pixels within a search area at time (T + ΔT), the sum of the differences in pixel values at each point is expressed by the following equation (1). Note that (-rx / 2 ≤ dx ≤ rx / 2, -ry / 2 ≤ jdx ≤ ry / 2). The tracking unit 16 only needs to find the dx and dy that minimize S(dx, dy).
[0041] The tracking unit 16 outputs the calculated information related to the movement amount (hereinafter referred to as "movement amount information") to the posture collapse detection unit 17. The movement amount information includes, for example, the average vector calculated by the tracking unit 16. The movement amount information may include, for example, each motion vector calculated by the tracking unit 16 in addition to the average vector. Note that information that makes it possible to identify which keypoint the motion vector corresponds to is assigned to each motion vector by the tracking unit 16.
[0042] The posture collapse detection unit 17 detects the posture collapse of the occupant based on the movement amount of the two-dimensional keypoints in the captured image calculated by the tracking unit 16, and the three-dimensional head position and face orientation of the occupant detected by the three-dimensional information detection unit 14. Here, since there are a plurality of keypoints, more specifically, the posture collapse detection unit 17 detects the posture collapse of the occupant based on the average of the movement amounts of the plurality of two-dimensional keypoints in the captured image calculated by the tracking unit 16.
[0043] For example, the posture collapse detection unit 17 detects the posture collapse of the occupant in accordance with a preset rule (hereinafter referred to as "posture collapse detection rule"). The posture collapse detection rule is set in advance by a developer or the like. As the posture collapse detection rule, for example, rules such as the following <Rule (1)> to <Rule (4)> are set. Note that in the first embodiment, for example, a large degree of inattentive driving by looking aside is also regarded as posture collapse.
[0044] <Rule (1)> When the three-dimensional head position of the occupant is outside a set range (hereinafter referred to as "first range"), the face orientation of the occupant is outside a set range (hereinafter referred to as "second range"), and the average of the movement amounts of the two-dimensional keypoints is equal to or greater than a set threshold (hereinafter referred to as "movement determination threshold"), it is detected that the occupant has lost their posture. <Rule (2)> When the three-dimensional head position of the occupant is undefined, the face orientation of the occupant is outside the second range, and the average of the movement amounts of the two-dimensional keypoints is equal to or greater than the movement determination threshold, it is detected that the occupant has lost their posture. <Rule (3)> When the three-dimensional head position of the occupant is within the first range, the average of the movement amounts of the two-dimensional keypoints is less than the movement determination threshold, and the face orientation of the occupant is undefined, it is detected that the occupant is looking aside significantly. <Rule (4)> When the three-dimensional head position and the face orientation of the occupant are undefined, and the average of the movement amounts of the two-dimensional keypoints is equal to or greater than the movement determination threshold, it is detected that the occupant has lost their posture.
[0045] Note that the posture loss detection rules as described above are merely examples. Appropriate rules may be set as the posture loss detection rules.
[0046] Further, for example, the posture loss detection unit 17 may detect the posture loss of the occupant using a machine learning model. The machine learning model is, for example, a model that receives as inputs information indicating the three-dimensional head position, information indicating the face orientation, and an average vector indicating the average of the movement amounts of the two-dimensional keypoints, and outputs information indicating whether there is a posture loss.
[0047] Here, Figures 5, 6, 7, and 8 illustrate the concept of detecting occupant posture collapse in Embodiment 1, based on an average vector showing the average of the occupant's face region, face orientation, and the amount of movement of the 2D keypoint in the captured image, as determined by the posture collapse detection unit 17. In the following explanation using Figures 5, 6, 7, or 8, the posture collapse detection unit 17 is assumed to detect occupant posture collapse according to the posture collapse detection rules described above. Also, for clarity, in Figures 5, 6, 7, and 8, captured images taken when the occupant is not experiencing posture collapse are shown together with captured images taken when the occupant is experiencing posture collapse. Figures 5A, 6A, 7A, and 8A are examples of captured images of an occupant in a normal posture state, without posture collapse. For example, Figure 5A shows an example of an image captured one frame before the image captured in Figure 5B, Figure 6A shows an example of an image captured one frame before the image captured in Figure 6B, Figure 7A shows an example of an image captured one frame before the image captured in Figure 7B, and Figure 8A shows an example of an image captured one frame before the image captured in Figure 8B. In Figures 5, 6, 7, and 8, the crew member is indicated by "D". In Figures 5, 6, 7, and 8, the face region is the area indicated by the dotted rectangle. Also, in Figures 5, 6, 7, and 8, the key point is indicated by a black circle. In Figures 5B, 6B, 7B, and 8B, the motion vector showing the amount of movement of each key point is indicated by a black arrow, and the average vector is indicated by a white arrow.
[0048] For example, Figure 5B shows an example of an image captured when an occupant is experiencing posture collapse, such as leaning against the door with their head or shoulders tilted sideways towards the door. In this case, for example, in the image captured in Figure 5B, the occupant's face region and keypoints are all captured, and the occupant's 3D head position is detected, but this 3D head position is outside the first range. The occupant's face orientation is also detected, but this face orientation is outside the second range. Furthermore, the average vector is above the motion determination threshold. In this case, the posture collapse detection unit 17 detects that the occupant is experiencing posture collapse, for example, according to the above-mentioned <Rule (1)>.
[0049] For example, Figure 6B shows an example of an image captured when the occupant is experiencing posture collapse, specifically leaning backward. In this case, for example, in the image captured in Figure 6B, the occupant's face region and keypoints are all captured, and the occupant's 3D head position is detected, but this 3D head position is outside the first range. The occupant's face orientation is also detected, but this face orientation is outside the second range. Furthermore, the average vector is above the motion determination threshold. In this case, the posture collapse detection unit 17 detects, for example, that the occupant is experiencing posture collapse according to the above-mentioned <Rule (1)>.
[0050] For example, Figure 7B shows an example of an image captured when the occupant is experiencing posture loss, specifically a forward-leaning posture. In this case, for example, in the image captured in Figure 7B, all of the occupant's key points are captured, but the occupant's face region is not captured, and the occupant's 3D head position and face orientation are not detected. Alternatively, the average vector is greater than or equal to the motion detection threshold. In this case, the posture loss detection unit 17 detects that the occupant is experiencing posture loss, for example, according to the above-mentioned <Rule (4)>.
[0051] For example, Figure 8B shows an example of an image captured when an occupant is leaning against a vehicle structure diagonally in front of them, causing their entire body to tilt diagonally forward, resulting in a posture collapse where the occupant is outside the field of view of the imaging device 2. In this case, for example, in the image captured in Figure 8B, the occupant's face region is not captured, and the occupant's 3D head position and face orientation are not detected. Also, regarding the occupant's key points, most are captured (in the example of Figure 8B, the top of the head, neck, and right shoulder), but some (in the example of Figure 8, the left shoulder) are not captured. The average vector, which shows the average of the motion vectors of the 2D key points corresponding to the captured key points, is greater than or equal to the motion determination threshold. In this case, the posture collapse detection unit 17 detects that the occupant is experiencing a posture collapse, for example, according to the above-mentioned <Rule (4)>.
[0052] In particular, in cases where some key points are not captured, as explained using Figure 8, techniques that detect posture collapse on the premise that two predetermined points of interest are detected may not be able to accurately detect that the occupant is experiencing posture collapse. For example, if the two points of interest are the right shoulder and the left shoulder, if one of them is not captured, it is not possible to distinguish whether the point of interest has moved out of the imaging device's field of view due to posture collapse, or whether one of the right and left shoulders has been temporarily obscured by an obstruction and therefore not captured in the image, even if there is little movement in either shoulder. In contrast, in the posture collapse detection device 1 according to Embodiment 1, the posture collapse detection unit 17 can detect that the occupant is experiencing posture collapse based on the average amount of movement of the two-dimensional key points, which are based on the occupant's key points that are being captured. As described above, the posture collapse detection unit 17 detects the occupant's posture collapse from the average amount of movement of the two-dimensional key points, so if even one of the key points among the top of the head, neck, right shoulder, and left shoulder is detected, it can detect that the occupant is experiencing posture collapse. In other words, in some cases, motion vectors cannot be determined (effective motion vectors cannot be determined), but in such cases, the tracking unit 16 calculates an average vector using only the motion vectors of the effective two-dimensional keypoints. This enables the posture collapse detection unit 17 to reliably detect posture collapse.
[0053] The posture collapse detection unit 17 can detect the occupant's posture collapse from the average (average vector) of the movement amounts in the images of the top of the head, neck, and left shoulder, even if the amount of movement (motion vector) in the image of the right shoulder is not calculated. Similarly, even if the amount of movement in the images of the right and left shoulders is not calculated, the posture collapse can be detected from the average of the movement amounts in the images of the top of the head and neck. The amount of movement of the two-dimensional keypoints that the posture collapse detection unit 17 considers when detecting the occupant's posture collapse is not fixedly determined. The posture collapse detection unit 17 can detect the occupant's posture collapse from the amount of movement of at least one two-dimensional keypoint if the amount of movement of that two-dimensional keypoint can be calculated.
[0054] Furthermore, in cases where the amount of movement of a 2D keypoint cannot be calculated for a given keypoint, it is possible that the tracking of the 2D keypoint position was not possible, or in other words, that the tracking unit 16 failed to track the 2D keypoint position. Here, we will explain an example of when the tracking unit 16 fails to track.
[0055] Figures 9 and 10 illustrate an example in Embodiment 1 where the tracking unit 16 fails to track.
[0056] Figure 9A shows an example of an image captured at time T. In Figure 9A, a certain two-dimensional keypoint is indicated by "401," and a small region template containing a reference pixel corresponding to the reference position, set with the two-dimensional keypoint position of the said two-dimensional keypoint as the reference position, is also indicated by "401." Figure 9B shows an example of an image captured at time (T + ΔT). For example, suppose that at this time, a certain keypoint corresponding to the two-dimensional keypoint indicated by "401" in Figure 9A moved in real space. Suppose that the movement of the said keypoint was not large enough to go outside the field of view of the imaging device 2, but moved to a certain extent. Furthermore, suppose that the said keypoint was temporarily obscured by an obstruction and therefore was not captured in the image. If it had not been obscured and had been captured, the said keypoint would have been captured as "404" in the image. In Figure 9B, for clarity, the keypoint that was not captured is shown. Furthermore, in Figure 9B, the search area is indicated by "403". For example, as shown in Figure 9B, if the amount of movement of a certain keypoint is large, and the position of that keypoint on the captured image, i.e., the amount of movement of the 2D keypoint on the captured image, is larger than the set search area, the tracking unit 16 will not be able to track correctly, or in other words, the tracking will fail. To prevent such failures, it is preferable to set the search area considering the amount of movement of the 2D keypoint between frames that can be expected on the captured image.
[0057] Figure 10A shows an example of an image captured at time T. The example image shown in Figure 10A is the same as the example image shown in Figure 9A. Figure 10B shows another example of an image captured at time (T + ΔT). For example, suppose that a key point corresponding to the aforementioned two-dimensional key point indicated by "401" in Figure 10A moves in real space. Suppose that the movement of this key point is large enough to move it out of the field of view of the imaging device 2. In this case, the key point will not be captured in the image, but in Figure 10B, for clarity, the key point that has moved out of the field of view of the imaging device 2 and is not captured is shown as "405". Also, in Figure 10B, the search area is indicated by "403". For example, for a keypoint that has moved so far that it is outside the field of view of the imaging device 2, as shown in Figure 10B, the tracking unit 16 cannot perform a correct tracking because the corresponding two-dimensional keypoint does not exist within the search area. In other words, the tracking fails. To prevent such failures, it is desirable that the field of view of the imaging device 2 be set considering the expected amount of keypoint movement.
[0058] As described above, even if a two-dimensional keypoint is not detected in the current captured image, the tracking unit 16 can set a search area and track the position of the two-dimensional keypoint by comparing the pixels included in the search area with reference pixels. This allows the attitude collapse detection unit 17 to calculate the amount of movement of the two-dimensional keypoint even if a two-dimensional keypoint is not detected in the current captured image. However, even so, as explained using Figures 9 and 10, for example, the tracking unit 16 may still fail to track the position of the two-dimensional keypoint.
[0059] Let's return to the explanation of the configuration example of the posture collapse detection device 1 using Figure 1. When the posture collapse detection unit 17 performs posture collapse detection processing to detect posture collapse in an occupant, it outputs information based on the posture collapse detection result (hereinafter referred to as "posture collapse detection information"). For example, when the posture collapse detection unit 17 detects that an occupant is experiencing posture collapse, it outputs posture collapse detection information to the output device to trigger an alarm. Alternatively, for example, if the vehicle is capable of autonomous driving, the posture collapse detection unit 17 may, for example, detect that an occupant is experiencing posture collapse and output posture collapse detection information to the autonomous driving control device to perform control such as stopping the vehicle or moving it to the side of the road.
[0060] The memory unit 18 stores various types of information. Here, as shown in Figure 1, the memory unit 18 is assumed to be provided in the posture collapse detection device 1, but this is only one example. The memory unit 18 may also be provided outside the posture collapse detection device 1, in a location accessible to the posture collapse detection device 1.
[0061] An example of operation of the posture collapse detection device 1 according to Embodiment 1 will be described. Figure 11 is a flowchart for explaining an example of operation of the posture collapse detection device 1 according to Embodiment 1. When the vehicle's power is turned on, for example, the posture collapse detection device 1 starts the operation shown in the flowchart of Figure 11, and after starting the operation shown in the flowchart of Figure 11, it repeats the operation shown in the flowchart of Figure 11 until, for example, the vehicle's power is turned off. However, the posture collapse detection device 1 only needs to perform the process of step ST2 once after starting operation.
[0062] The image acquisition unit 11 acquires captured images from the imaging device 2 (step ST1). The image acquisition unit 11 outputs the acquired captured images to the calibration unit 12, the face area detection unit 13, or the key point detection unit 15. The image acquisition unit 11 also stores the acquired captured images in the storage unit 18 in chronological order, associating them with the date and time the images were acquired.
[0063] The calibration unit 12 performs calibration of a three-dimensional model of a person's head, which is used to detect the occupant's three-dimensional head position and face orientation, based on the captured images acquired by the image acquisition unit 11 during the calibration period. Until the calibration unit 12 completes the calibration (if "NO" is selected in step ST2), the posture collapse detection device 1 suspends processing from step ST3 onward. Once the calibration unit 12 completes the calibration (if "YES" is selected in step ST2), the calibration unit 12 outputs calibration completion information to the face region detection unit 13.
[0064] Then, the face region detection unit 13 detects the occupant's face region based on the captured image acquired by the image acquisition unit 11 in step ST1 (step ST3). The face region detection unit 13 outputs information indicating the detected face region (hereinafter referred to as "face region information") to the 3D information detection unit 14.
[0065] In step ST3, the 3D information detection unit 14 detects the occupant's 3D head position and face orientation based on the face region detection unit 13 and the 3D model (step ST4). The 3D information detection unit 14 outputs 3D detection information indicating the detected occupant's 3D head position and face orientation to the keypoint detection unit 15. The 3D information detection unit 14 also stores the 3D detection information in the storage unit 18, associating it with the captured image that was used to detect the occupant's 3D head position and face orientation.
[0066] The keypoint detection unit 15 detects points corresponding to the occupant's keypoints (hereinafter referred to as "2D keypoints") on the captured image based on the image acquired by the image acquisition unit 11 in step ST1 (step ST5). When the keypoint detection unit 15 detects a 2D keypoint, it outputs the 2D keypoint information to the tracking unit 16. The keypoint detection unit 15 also stores the 2D keypoint information in the storage unit 18, associating it with the captured image that was used to detect the occupant's 3D head position and face orientation.
[0067] The tracking unit 16 tracks the 2D keypoint positions detected by the keypoint detection unit 15 in a time series (step ST6). The tracking unit 16 then calculates the amount of movement of the 2D keypoints in the captured image. The tracking unit 16 also calculates the average of the amount of movement of each 2D keypoint in the captured image. The tracking unit 16 outputs the amount of movement information to the posture collapse detection unit 17.
[0068] The posture collapse detection unit 17 detects the occupant's posture collapse (step ST7) based on the amount of movement in the captured images of the two-dimensional keypoints calculated by the tracking unit 16 in step ST6, and the occupant's three-dimensional head position and face orientation detected by the three-dimensional information detection unit 14 in step ST4. More specifically, the posture collapse detection unit 17 detects the occupant's posture collapse based on the average of the amount of movement in the captured images of multiple two-dimensional keypoints calculated by the tracking unit 16. When the posture collapse detection unit 17 has performed the posture collapse detection processing to detect the occupant's posture collapse, it outputs posture collapse detection information.
[0069] In the above embodiment 1, if the face region detection unit 13 fails to detect the occupant's face region based on the captured image acquired by the image acquisition unit 11, it does not immediately detect that the face region could not be detected, or in other words, that the face region is undefined. Instead, for example, it may perform a buffering process (hereinafter referred to as "face region buffering process") and detect that the face region is undefined if face detection fails for a certain percentage of a certain length in the time direction (buffer size). The buffer size is predetermined by the developer, etc. If the percentage does not exceed a certain percentage, the face region detection unit 13 outputs face region information relating to the previously detected face region to the 3D information detection unit 14. The 3D information detection unit 14 then uses this face region information to detect the occupant's 3D head position and face orientation. This enables the next process, keypoint detection, to be executed stably. The face region buffering process is performed in the face region detection process in step ST3 of Figure 11 described above. For example, in step ST3, the face region detection unit 13 adds to the buffer value if it is unable to detect the occupant's face region based on the captured image acquired by the image acquisition unit 11 in step ST1. On the other hand, the face region detection unit 13 subtracts from the buffer value if it is able to detect the occupant's face region. The face region detection unit 13 then detects that the face region is undefined if the buffer value exceeds a predetermined threshold.
[0070] Furthermore, in the above embodiment 1, if the keypoint detection unit 15 fails to detect a two-dimensional keypoint based on the captured image acquired by the image acquisition unit 11, it may not immediately detect that the two-dimensional keypoint is undefined, but rather perform a buffering process (hereinafter referred to as "keypoint buffering process") to detect that the two-dimensional keypoint is undefined if the number of undetected two-dimensional keypoints exceeds a certain percentage within a certain length in the time direction (buffer size). The buffer size is predetermined by the developer, etc. If the number of undetected two-dimensional keypoints does not exceed a certain percentage, the keypoint detection unit 15 outputs two-dimensional keypoint information indicating the previously detected two-dimensional keypoint to the tracking unit 16. The tracking unit 16 then uses this two-dimensional keypoint information to track the position of the two-dimensional keypoint. This enables the next process, tracking the position of the two-dimensional keypoint, to be executed stably. The keypoint buffering process is performed in the keypoint detection process in step ST5 of Figure 11 described above. For example, in step ST5, the keypoint detection unit 15 adds to the buffer value if it is unable to detect a two-dimensional keypoint based on the captured image acquired by the image acquisition unit 11 in step ST1. On the other hand, the keypoint detection unit 15 subtracts from the buffer value if it is able to detect a two-dimensional keypoint. The keypoint detection unit 15 then detects that the two-dimensional keypoint is undefined if the buffer value exceeds a predetermined threshold.
[0071] Furthermore, in the above embodiment 1, the 3D information detection unit 14 was configured to detect both the occupant's 3D head position and face orientation, but this is merely one example. The 3D information detection unit 14 may be configured to detect either the occupant's 3D head position or face orientation.
[0072] Furthermore, in the above embodiment 1, the posture collapse detection device 1 is provided with a calibration unit 12, but this is merely an example, and it is not essential for the posture collapse detection device 1 to include a calibration unit 12. However, by providing the posture collapse detection device 1 with a calibration unit 12, as described above, the 3D model can be made to closely resemble the shape of the occupant's face. If the posture collapse detection device 1 is configured without a calibration unit 12, the process of step ST2 can be omitted in the operation of the posture collapse detection device 1 as described using the flowchart in Figure 11.
[0073] Furthermore, in the above embodiment 1, the posture collapse detection device 1 is provided with a face region detection unit 13 and a 3D information detection unit 14, but this is merely an example, and the posture collapse detection device 1 is not required to include a face region detection unit 13 and a 3D information detection unit 14. For example, if the posture collapse detection device 1 is configured without a face region detection unit 13 and a 3D information detection unit 14, the key point detection unit 15 may detect 2D key points from the captured image using known image recognition technology or the like. For example, the key point detection unit 15 may detect 2D key points using a machine learning model that takes the captured image as input and outputs information indicating key points on the captured image. However, the posture collapse detection device 1 can more accurately detect 2D key points by providing a face region detection unit 13 and a 3D information detection unit 14 and detecting 2D key points based on the occupant's 3D head position or face orientation. For example, if distortion occurs in the captured image, the image recognition process may experience detection errors in 2D key points due to the effects of the distortion. Furthermore, if the posture collapse detection device 1 is configured without a face region detection unit 13 and a three-dimensional information detection unit 14, the posture collapse detection unit 17 only needs to detect the occupant's posture collapse from the amount of movement in the captured image of the two-dimensional keypoint calculated by the tracking unit 16. However, the posture collapse detection unit 17 can detect the occupant's posture collapse with greater accuracy by also using the occupant's three-dimensional head position or face orientation as a posture collapse detection element. For example, by using the occupant's face orientation as a posture collapse detection element, the posture collapse detection unit 17 can detect occupant posture collapses that are difficult to detect from the amount of movement in the captured image of the two-dimensional keypoint alone, such as excessive distraction. If the posture collapse detection device 1 is configured without a face region detection unit 13 and a three-dimensional information detection unit 14, the processing of steps ST3 to ST4 can be omitted in the operation of the posture collapse detection device 1 as described using the flowchart in Figure 11.
[0074] Furthermore, in the above embodiment 1, the face region detection unit 13 performs face region detection processing after calibration by the calibration unit 12 is completed, but this is merely an example. For example, face region detection processing by the face region detection unit 13 may be performed in parallel with calibration by the calibration unit 12. In this case, in the operation of the posture collapse detection device 1 as described using the flowchart in Figure 11, the processing of steps ST1 to ST2 and the processing of steps ST1, ST3 to ST7 are performed in parallel.
[0075] Furthermore, in the above embodiment 1, it was assumed that the keypoint detection unit 15 performs two-dimensional keypoint detection processing for the captured image acquired by the image acquisition unit 11 every frame, but this is merely an example. For example, the keypoint detection unit 15 may perform two-dimensional keypoint detection in the captured image only once after the posture collapse detection device 1 starts operating, or periodically. In this case, the tracking unit 16 may, for example, refer to the storage unit 18 and track the two-dimensional keypoint position by matching a small-region template set based on the latest stored two-dimensional keypoint information with a search area set on the captured image of the current frame. Also, in the above embodiment 1, for example, even if the keypoint detection unit 15 can detect two-dimensional keypoints in the captured image acquired by the image acquisition unit 11, the tracking unit 16 may track the two-dimensional keypoint position by matching the small-region template with the search area.
[0076] Furthermore, in the above embodiment 1, the posture collapse detection device 1 is an in-vehicle device mounted on a vehicle, and the image acquisition unit 11, calibration unit 12, face region detection unit 13, 3D information detection unit 14, key point detection unit 15, tracking unit 16, and posture collapse detection unit 17 are provided on the in-vehicle device. However, the system may be configured such that some of the image acquisition unit 11, calibration unit 12, face region detection unit 13, 3D information detection unit 14, key point detection unit 15, tracking unit 16, and posture collapse detection unit 17 are mounted on the in-vehicle device of the vehicle, and the others are provided on a server connected to the in-vehicle device via a network, with the in-vehicle device and the server forming the system. Alternatively, the image acquisition unit 11, calibration unit 12, face region detection unit 13, 3D information detection unit 14, key point detection unit 15, tracking unit 16, and posture collapse detection unit 17 are all provided on the server.
[0077] Figures 12A and 12B show an example of the hardware configuration of the posture collapse detection device 1 according to Embodiment 1. In Embodiment 1, the functions of the image acquisition unit 11, the calibration unit 12, the face region detection unit 13, the three-dimensional information detection unit 14, the key point detection unit 15, the tracking unit 16, and the posture collapse detection unit 17 are realized by the processing circuit 101. That is, the posture collapse detection device 1 includes a processing circuit 101 for controlling the detection of posture collapse of an occupant based on an captured image in which at least the face of the occupant present in the vehicle interior is captured. The processing circuit 101 may be dedicated hardware as shown in Figure 12A, or it may be a processor 104 that executes a program stored in memory as shown in Figure 12B.
[0078] If the processing circuit 101 is dedicated hardware, it may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof.
[0079] When the processing circuit is a processor 104, the functions of the image acquisition unit 11, the calibration unit 12, the face region detection unit 13, the three-dimensional information detection unit 14, the key point detection unit 15, the tracking unit 16, and the posture collapse detection unit 17 are realized by software, firmware, or a combination of software and firmware. The software or firmware is written as a program and stored in memory 105. The processor 104 reads and executes the program stored in memory 105 to perform the functions of the image acquisition unit 11, the calibration unit 12, the face region detection unit 13, the three-dimensional information detection unit 14, the key point detection unit 15, the tracking unit 16, and the posture collapse detection unit 17. In other words, the posture collapse detection device 1 includes memory 105 for storing a program that, when executed by the processor 104, will result in the execution of steps ST1 to ST7 in Figure 11 described above. Furthermore, the program stored in memory 105 can be said to cause the computer to execute the processing procedures or methods of the image acquisition unit 11, the calibration unit 12, the face region detection unit 13, the three-dimensional information detection unit 14, the key point detection unit 15, the tracking unit 16, and the posture collapse detection unit 17. Here, memory 105 refers to non-volatile or volatile semiconductor memory such as RAM, ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), or magnetic disks, flexible disks, optical disks, compact disks, minidiscs, DVDs (Digital Versatile Discs), etc.
[0080] Furthermore, the functions of the image acquisition unit 11, calibration unit 12, face region detection unit 13, 3D information detection unit 14, key point detection unit 15, tracking unit 16, and posture collapse detection unit 17 may be partially implemented by dedicated hardware and partially by software or firmware. For example, the image acquisition unit 11 can be implemented by a processing circuit 101 as dedicated hardware, while the calibration unit 12, face region detection unit 13, 3D information detection unit 14, key point detection unit 15, tracking unit 16, and posture collapse detection unit 17 can be implemented by a processor 104 reading and executing a program stored in memory 105. The storage unit 18 is composed of memory 105, etc. The posture collapse detection device 1 also includes an imaging device 2, an output device (not shown), or an automatic driving control device (not shown), and an input interface device 102 and an output interface device 103 that perform wired or wireless communication.
[0081] As described above, according to Embodiment 1, the posture collapse detection device 1 is configured to include: an image acquisition unit 11 that acquires captured images in a time series in which at least the faces of occupants in the vehicle interior are captured; a key point detection unit 15 that detects two-dimensional key points corresponding to key points, which are points indicating the joints or facial parts of the occupants, on the captured images based on the captured images acquired by the image acquisition unit 11; a tracking unit 16 that tracks the two-dimensional key point positions, which are the positions of the two-dimensional key points detected by the key point detection unit 15 on the captured images, in a time series and calculates the amount of movement of the two-dimensional key points in the captured images; and a posture collapse detection unit 17 that detects the posture collapse of the occupants based on the amount of movement of the two-dimensional key points in the captured images calculated by the tracking unit 16. Therefore, the posture collapse detection device 1 can detect the posture collapse of vehicle occupants with greater accuracy than conventional technologies that detect posture collapse on the premise that two predetermined points of interest are detected. The posture collapse detection device 1 can accurately detect posture collapse even if, for example, the occupant's head, right shoulder, or left shoulder goes outside the field of view of the imaging device 2 due to the occupant's posture collapse, or is temporarily obscured by an obstruction, or the occupant's face is temporarily turned far away from the direction of the imaging device 2 and the face is not captured.
[0082] Furthermore, in the posture collapse detection device 1, the tracking unit 16 may set a small region template that includes a reference pixel, which is a pixel corresponding to a 2D keypoint detected by the keypoint detection unit 15 based on past captured images, using the 2D keypoint position of the 2D keypoint detected by the keypoint detection unit 15 as the reference position. On the captured image acquired by the image acquisition unit 11, a search area wider than the small region template is set, expanded vertically and horizontally by the amount of pixels used for setting the search area, centered on the reference pixel. The tracking unit then tracks the 2D keypoint position by comparing the pixels included in the search area with the pixels included in the small region template. This allows the posture collapse detection device 1 to track the 2D keypoint position without interrupting the tracking, even if, for example, a 2D keypoint could not be detected on the current captured image.
[0083] Furthermore, the posture collapse detection device 1 may be configured to detect the occupant's posture collapse based on the amount of movement of the 2D keypoint in the captured image calculated by the tracking unit 16 and the occupant's 3D head position or face orientation based on the face region detected by the face region detection unit 13 and a 3D model of the human head. The posture collapse detection unit 17 may also be configured to detect the occupant's posture collapse based on the amount of movement of the 2D keypoint in the captured image calculated by the tracking unit 16 and the occupant's 3D head position or face orientation detected by the 3D information detection unit 14. This allows the posture collapse detection device 1 to detect the occupant's posture collapse with greater accuracy. For example, by including the occupant's face orientation as a posture collapse detection element, the posture collapse detection device 1 can detect occupant posture collapses that are difficult to detect from the amount of movement of the 2D keypoint in the captured image alone, such as excessive distraction.
[0084] Furthermore, the posture collapse detection device 1 may be configured to detect 2D keypoints by including a face region detection unit 13 that detects the face region in which the occupant's face is captured based on the captured image acquired by the image acquisition unit 11, and a 3D information detection unit 14 that detects the occupant's 3D head position based on the face region detected by the face region detection unit 13 and a 3D model of the human head, and a keypoint detection unit 15 that calculates the relative positional relationship between the occupant's 3D head position and the 3D position of the keypoint according to a 3D keypoint estimation rule based on the occupant's 3D head position detected by the 3D information detection unit 14, tentatively determines the position of the keypoint on the captured image based on the calculated relative positional relationship, and adjusts the position of the tentatively determined keypoint on the captured image by applying a scaling ratio obtained from the size of the occupant's face region on the captured image. This allows the posture collapse detection device 1 to detect 2D keypoints more accurately, taking into account, for example, the effects of distortion occurring in the captured image.
[0085] Furthermore, in the posture collapse detection device 1, the 3D information detection unit 14 detects the occupant's 3D head position and face orientation, and the key point detection unit 15 calculates the relative positional relationship between the occupant's 3D head position and the 3D position of the key point according to the occupant's face orientation, based on the occupant's 3D head position and face orientation detected by the 3D information detection unit 14, according to the 3D key point estimation rules, and then provisionally determines the position of the key point on the captured image based on the calculated relative positional relationship, and adjusts the position of the provisionally determined key point on the captured image by applying a scaling ratio obtained from the size of the occupant's face region on the captured image, thereby detecting the 2D key point. This allows the posture collapse detection device 1 to detect 2D key points more accurately, taking into account, for example, the effects of distortion occurring in the captured image.
[0086] Furthermore, the posture collapse detection device 1 may include a calibration unit 12 that performs calibration of the 3D model based on the captured images acquired by the image acquisition unit 11 during the calibration period, and the 3D information detection unit 14 may be configured to detect the occupant's 3D head position or face orientation based on the face region detected by the face region detection unit 13 and the 3D model calibrated by the calibration unit 12. This allows the posture collapse detection device 1 to make the 3D model closer to the occupant's face shape. As a result, the posture collapse detection device 1 can improve the accuracy of detecting the occupant's 3D head position or face orientation using the 3D model.
[0087] Furthermore, any component of the embodiment can be modified, or any component of the embodiment can be omitted.
[0088] The posture collapse detection device described herein can accurately detect posture collapse of occupants present inside a vehicle.
[0089] 1. Posture collapse detection device, 2. Imaging device, 11. Image acquisition unit, 12. Calibration unit, 13. Face region detection unit, 14. Three-dimensional information detection unit, 15. Key point detection unit, 16. Tracking unit, 17. Posture collapse detection unit, 18. Storage unit, 101. Processing circuit, 102. Input interface device, 103. Output interface device, 104. Processor, 105. Memory.
Claims
1. A posture collapse detection device comprising: an image acquisition unit that acquires a series of images of at least the faces of occupants inside a vehicle; a key point detection unit that detects two-dimensional key points corresponding to key points that indicate joints or parts of the occupant's face on the captured images based on the images acquired by the image acquisition unit; a tracking unit that tracks the two-dimensional key point positions, which are the positions of the two-dimensional key points detected by the key point detection unit on the captured images, in a series of times and calculates the amount of movement of the two-dimensional key points in the captured images; and a posture collapse detection unit that detects posture collapse of the occupant based on the amount of movement of the two-dimensional key points in the captured images calculated by the tracking unit.
2. The posture collapse detection device according to claim 1, characterized in that the key point detection unit detects a plurality of two-dimensional key points, the tracking unit tracks the position of each two-dimensional key point, calculates the average of the movement amounts of the plurality of two-dimensional key points from the movement amount in the captured image for each two-dimensional key point, and the posture collapse detection unit detects the posture collapse of the occupant based on the average of the movement amounts in the captured image of the plurality of two-dimensional key points calculated by the tracking unit.
3. The posture collapse detection device according to claim 1 or 2, characterized in that the tracking unit sets a small region template including a reference pixel which is a pixel corresponding to the reference position of the two-dimensional key point that has been detected by the key point detection unit based on past captured images, using the two-dimensional key point position of the two-dimensional key point as the reference position, sets a search region wider than the small region template by extending the vertical and horizontal directions by the amount of pixels for setting the search region, centered on the reference pixel, on the captured image acquired by the image acquisition unit, and tracks the position of the two-dimensional key point by comparing the pixels included in the search region with the pixels included in the small region template.
4. The posture collapse detection device according to claim 1, comprising: a face region detection unit that detects a face region in which the occupant's face is captured based on the captured image acquired by the image acquisition unit; and a three-dimensional information detection unit that detects the occupant's three-dimensional head position or face orientation based on the face region detected by the face region detection unit and a three-dimensional model of a human head, wherein the posture collapse detection unit detects the occupant's posture collapse based on the amount of movement of the two-dimensional keypoint in the captured image calculated by the tracking unit and the occupant's three-dimensional head position or face orientation detected by the three-dimensional information detection unit.
5. The posture collapse detection device according to claim 1, comprising: a face region detection unit that detects a face region in which the occupant's face is captured based on the captured image acquired by the image acquisition unit; and a three-dimensional information detection unit that detects the occupant's three-dimensional head position based on the face region detected by the face region detection unit and a three-dimensional model of a human head, wherein the key point detection unit calculates the relative positional relationship between the occupant's three-dimensional head position and the three-dimensional position of the key point according to a three-dimensional key point estimation rule based on the three-dimensional head position of the occupant detected by the three-dimensional information detection unit, provisionally determines the position of the key point on the captured image based on the calculated relative positional relationship, and detects the two-dimensional key point by adjusting the provisionally determined position of the key point on the captured image by applying a scaling ratio obtained from the size of the occupant's face region on the captured image.
6. The posture collapse detection device according to claim 5, characterized in that the three-dimensional information detection unit detects the three-dimensional head position and face orientation of the occupant, the key point detection unit calculates the relative positional relationship between the occupant's three-dimensional head position and the three-dimensional position of the key point according to the three-dimensional key point estimation rule based on the occupant's face orientation, and provisionally determines the position of the key point on the captured image based on the calculated relative positional relationship.
7. The posture collapse detection device according to claim 4 or 6, further comprising a calibration unit that performs calibration of the three-dimensional model based on the captured images acquired by the image acquisition unit during a calibration period, wherein the three-dimensional information detection unit detects the three-dimensional head position or face orientation of the occupant based on the face region detected by the face region detection unit and the three-dimensional model calibrated by the calibration unit.
8. A posture collapse detection method comprising: an image acquisition unit acquiring captured images in which at least the faces of occupants in the vehicle interior are captured in a time series; a key point detection unit detecting two-dimensional key points corresponding to key points that indicate joints or parts of the occupant's face on the captured images based on the captured images acquired by the image acquisition unit; a tracking unit tracking the two-dimensional key point positions, which are the positions of the two-dimensional key points detected by the key point detection unit on the captured images, in a time series and calculating the amount of movement of the two-dimensional key points in the captured images; and a posture collapse detection unit detecting posture collapse of the occupant based on the amount of movement of the two-dimensional key points in the captured images calculated by the tracking unit.
