Behavior propriety determination system and behavior propriety determination method

Abstract

The present invention appropriately determines whether a subject's current behavior is appropriate by estimating the subject's current behavior and state of attention to the surroundings.　A behavior propriety determination system according to the present invention comprises: a behavior estimation unit that, on the basis of sensor information from a first sensor device provided in a target space, estimates a current behavior, which is a behavior currently being performed by a subject; an attention estimation unit that detects the line-of-sight direction of the subject on the basis of sensor information from a second sensor device provided in the target space, and estimates an attention target, which is a target to which the subject is directing attention, or a non-attention target, which is a target to which the subject is not directing attention; and a propriety determination unit that determines whether the subject's current behavior is appropriate on the basis of the subject's current behavior and attention target or non-attention target.

Description

Action quality judgment system and action quality judgment method

[0001] The present invention relates to an action quality judgment system and an action quality judgment method.

[0002] Patent Document 1 discloses a technology that uses a trained motion recognition model to recognize the manual work of a subject as a connection of a plurality of local motions from the movement of the three-dimensional skeleton of the subject, and provides improvement proposals for local motions whose recognition accuracy in the recognition is less than a threshold. Patent Document 2 discloses a driving support device that uses, as an evaluation value, the ratio of the viewing angle of a preceding vehicle as seen from the driver to the angular variance value of the driver's line-of-sight direction, relaxes the forward warning more than usual when the evaluation value is equal to or higher than a threshold, and sets the forward warning to a normal warning when the evaluation value is less than the threshold.

[0003] Japanese Unexamined Patent Application Publication No. 2023-129230 Japanese Unexamined Patent Application Publication No. 2006-318049

[0004] The present disclosure provides an action quality judgment system and an action quality judgment method capable of estimating the current action and the state of attention to the surroundings of a subject who is a monitoring target, and appropriately judging whether or not the current action of the subject is appropriate.

[0005] One aspect of the behavioral goodness / badness judgment system in this disclosure includes: a behavioral estimation unit that estimates the current behavior of a person in a target space based on sensor information from a first sensor device provided in the target space; an attention estimation unit that detects the direction of the subject's gaze based on sensor information from a second sensor device provided in the target space and estimates an object of attention that the subject is paying attention to or an object of non-attention that the subject is not paying attention to; and a goodness / badness judgment unit that determines whether the subject's current behavior is good or bad based on the subject's current behavior and the object of attention or non-attention. Furthermore, one aspect of the behavioral goodness judgment method in this disclosure includes: a behavioral estimation step in which a computer estimates the current behavior of a person in a target space based on sensor information from a first sensor device provided in the target space; an attention estimation step in which a computer detects the direction of the subject's gaze based on sensor information from a second sensor device provided in the target space and estimates the attention target or the non-attention target that the subject is paying attention to; and a goodness judgment step in which the computer determines whether the subject's current behavior is good or bad based on the subject's current behavior and the attention target or non-attention target. This specification includes all the contents of Japanese Patent Application No. 2025-048839, filed on March 24, 2025.

[0006] The behavioral appropriateness judgment system and behavioral appropriateness judgment method described herein can appropriately determine whether the current behavior of a person being monitored is appropriate or not by estimating the person's current behavior and level of attention to their surroundings.

[0007] Figure 1 shows a schematic overview of the overall configuration of the behavioral compliance judgment system according to the embodiment. Figure 2 shows a schematic configuration of the monitoring device that constitutes the behavioral compliance judgment system. Figure 3 shows an example of related information. Figure 4 is a flowchart showing the operation procedure of the behavioral compliance judgment system.

[0008] (Knowledge and other information forming the basis of this disclosure) At the time the inventors conceived this disclosure, there was a technology that used a trained motion recognition model to recognize the manual work of a subject as a series of local movements from the movement of the subject's three-dimensional skeleton, and to suggest improvements for local movements whose recognition accuracy in the above recognition was below a threshold. There was also a technology that disclosed a driver assistance device in which the ratio of the field of view angle of the preceding vehicle as seen by the driver to the angular dispersion value of the driver's line of sight direction was used as an evaluation value, and if the evaluation value was above a threshold, the forward warning was made more lenient than usual, and if the evaluation value was below the threshold, the forward warning was made a normal warning. However, in the above technology that uses the recognition of local movements, it is possible to determine the absence of some local movements necessary for manual work, but the attention of the subject in movements that are not missing is not detected, so there may be cases where dangerous movements in manual work cannot be detected. In addition, in the above technology that uses the angular dispersion value of the driver's line of sight direction, the object that the driver should focus on is limited to the preceding vehicle, and the state of attention to other objects such as pedestrians and oncoming vehicles is not considered, so there may be cases where the quality of the driver's driving behavior cannot be appropriately judged. The inventors discovered the above-mentioned problem and, in order to solve it, have come to form the subject matter of this disclosure. Accordingly, this disclosure provides a behavioral

[0009] The embodiments will be described in detail below with reference to the drawings. However, some descriptions may be omitted to avoid unnecessary detail. For example, detailed explanations of already well-known matters or redundant explanations of substantially identical configurations may be omitted. The accompanying drawings and the following description are provided to enable those skilled in the art to fully understand this disclosure and are not intended to limit the subject matter described in the claims.

[0010] (Embodiment) The embodiment will be described below with reference to Figures 1 to 4. [1. Configuration of the Behavior Appropriateness Judgment System] Figure 1 is a diagram showing the configuration of the Behavior Appropriateness Judgment System 1 according to the embodiment. The Behavior Appropriateness Judgment System 1 considers a person in the target space S as the subject P and determines whether the subject P's current behavior is good or not based on the subject P's current behavior and the object of attention. Here, the judgment of whether the behavior is good or not means the judgment of whether it is a behavior for which it is considered necessary to issue some kind of warning, for example, whether it is a dangerous behavior or not, or whether it is an appropriate behavior that does not cause any problems.

[0011] The behavioral appropriateness judgment system 1 includes a monitoring device 2. The monitoring device 2 processes information based on sensor data from at least one first sensor device 3 and at least one second sensor device 4. The at least one first sensor device 3 and the at least one second sensor device 4 may be included in the behavioral appropriateness judgment system 1. The first sensor device 3 and the second sensor device 4 are provided in the target space S. The target space S is, for example, a living space in a building, but it may also be a personal office in an office space.

[0012] The first sensor device 3 transmits sensor information that can be used to detect the movement of the subject P to the monitoring device 2 as sensor information. The first sensor device 3 may be, for example, a visible light camera, an infrared camera, an infrared array sensor, or a pyroelectric infrared sensor. Alternatively, the first sensor device 3 may be a device equipped with a wireless transceiver that detects human movement by Wi-Fi® sensing.

[0013] In this embodiment, the first sensor device 3 is a visible light camera and is installed on electrical equipment such as outlets C1 and C2, switches SW, and lighting fixtures L, which are located on the walls and ceiling of the target space S.

[0014] The second sensor device 4 transmits sensor information that can be used to detect the direction of the subject P's line of sight to the monitoring device. The sensor information that can be used to detect the direction of the line of sight is, for example, information necessary for detecting the line of sight intersection, which is the point where the surface of a real object in the vicinity of the subject P intersects with the subject P's line of sight, and may include images of the subject P's eyes, images of their head, information on the movement of their eyeballs, etc. The second sensor device 4 may be, for example, a visible light camera, an infrared camera, or an infrared array sensor. Alternatively, the second sensor device 4 may be a device equipped with a wireless transceiver that detects the movement and posture of a person's head by Wi-Fi sensing.

[0015] Alternatively, the second sensor device 4 may be, for example, a goggle-type or glasses-type eye-tracking sensor. With this configuration, it is possible to easily identify the attention-grabbing or non-attention-grabbing objects of the subject P, described later, using a goggle-type or glasses-type eye-tracking sensor that is easy to wear during work. The goggle-type or glasses-type eye-tracking sensor may, for example, acquire an image of the subject P's eyes, or detect eye movement using MEMS (Micro Electro Mechanical Systems).

[0016] In this embodiment, the second sensor device 4 is a visible light camera and is installed on an electrical outlet C3, which is an electrical fixture installed on the wall of the target space S.

[0017] The monitoring device 2 determines whether the subject P's behavior is good or not based on sensor information from the first sensor device 3 and the second sensor device 4.

[0018] [2. Configuration of the Monitoring Device] Figure 2 shows the configuration of the monitoring device 2. The monitoring device 2 is a computer comprising a processor 10 and a memory 11. The memory 11 is composed of, for example, a volatile and / or non-volatile semiconductor memory. Relevance information 12 is pre-stored in the memory 11. Relevance information 12 will be described later.

[0019] The processor 10 is, for example, a arithmetic processing unit such as a CPU. The processor 10 may have a configuration that includes a ROM on which the program is written, a RAM for temporary data storage, etc. The processor 10 has, as functional elements or functional units, an action estimation unit 14, an attention estimation unit 15, a good / bad judgment unit 16, and a notification unit 17.

[0020] These functional elements of the processor 10 are realized, for example, by the processor 10 of the monitoring device 2 (which is a computer) executing a program 13 stored in the memory 11. The program 13 can be stored in any storage medium that is readable by a computer. Alternatively, all or part of the above functional elements of the processor 10 can be configured by hardware, each including one or more electronic circuit components.

[0021] The behavior estimation unit 14 estimates the current behavior of the subject P based on sensor information from the first sensor device 3. This estimation of the current behavior may be repeated at predetermined time intervals. For example, the behavior estimation unit 14 may recognize the movement of the subject P's skeleton based on sensor information from the first sensor device 3 at predetermined time intervals and estimate the subject P's current behavior from the movement of the skeleton. As an example, the behavior estimation unit 14 may acquire an image of the subject P as sensor information from the first sensor device 3, which is a visible light camera, recognize the movement of the subject P's skeleton based on the acquired image of the subject P, and estimate the subject P's current behavior from that movement of the skeleton. With this configuration, the subject P's current behavior can be estimated while also considering the subject P's privacy.

[0022] The recognition of skeletal movement from images of subject P can be performed, for example, using existing posture estimation AI or skeletal estimation libraries that can detect human skeletal information in real time from camera images (moving images), such as MMPose (Multi-Person Pose Estimation), in accordance with conventional technology. Alternatively, when a Wi-Fi sensing device is used as the first sensor device 3, the recognition of skeletal movement of subject P can be performed using existing technologies such as WiPose, which performs 3D posture estimation from Wi-Fi radio waves. In this case, the skeletal movement of subject P can be recognized even if there is an optical obstruction between the first sensor device 3 and subject P.

[0023] Furthermore, the estimation of the subject P's current behavior from skeletal movement can be performed, for example, by using a pre-trained model that has been trained using machine learning such as deep learning to recognize the relationship between specific movements or actions and skeletal movement.

[0024] The attention estimation unit 15 recognizes the orientation of the subject P's face based on sensor information from the second sensor device 4, detects the subject P's gaze direction, and estimates whether the object the subject P is paying attention to is an attention object or an attention object the subject P is not paying attention to. With this configuration, the attention object or attention object of the subject P can be objectively determined from the direction of the subject P's gaze.

[0025] The above estimation of items requiring attention and items not requiring attention may be output as, for example, a list of items requiring attention and a list of items not requiring attention estimated in a predetermined time period in the most recent past (for example, the last 5 minutes), and these lists may be continuously updated and output at predetermined time intervals as time progresses.

[0026] The attention estimation unit 15, for example, in accordance with prior art, detects the line of sight intersection point, which is the point where the surface of a real object in the vicinity of subject P intersects with subject P's line of sight, from an image of subject P's eyes from a second sensor device 4, which is a visible light camera, and an image of subject P's surroundings. Instead of the eye image, the direction of subject P's line of sight may be estimated using skeletal information of the head that can be obtained from the image of subject P. The attention estimation unit 15 can estimate the direction of the face recognized from the skeletal structure of the head as the line of sight, and estimate the line of sight intersection point from the estimated line of sight.

[0027] Furthermore, the attention estimation unit 15 can estimate that objects in the vicinity of the subject P that the point of line of sight lingers for a predetermined period of time or longer are objects of the subject P's attention. In addition, the attention estimation unit 15 can estimate that objects in the vicinity of the subject P that the point of line of sight did not linger (for example, objects that the line of sight simply passed over) and objects that the point of line of sight lingered for less than the predetermined period of time are objects that the subject P did not pay attention to.

[0028] The good / bad judgment unit 16 determines whether the subject P's current behavior is good or bad based on the subject P's current behavior estimated by the behavior estimation unit 14 and the subject P's attention target or non-attention target estimated by the attention estimation unit 15. With this configuration, the subject P's current behavior and attention status to the surroundings are objectively grasped as current behavior and attention target or non-attention target based on sensor information from the first sensor device 3 and second sensor device 4 provided in the target space S, so that it is possible to appropriately determine whether the subject P's current behavior is good or bad.

[0029] For example, the good / bad judgment unit 16 refers to relationship information 12, which associates a person's actions with related objects (real objects related to those actions) and the degree of relationship between those actions and the related objects, to determine whether the current actions of the subject P are good or bad. The relationship information 12 is predetermined and stored in memory 11.

[0030] Figure 3 shows an example of relevance information 12. In the example in Figure 3, the leftmost column (column 1) shows the actions, and the second column to its right lists the related real-world objects for each action. The third column to the right of the second column shows the degree of relevance for each related object, which is the degree of relevance to the corresponding action, as a five-point rating scale. The five-point rating scale may, for example, indicate the relative level of relevance between the related objects and the actions.

[0031] The quality judgment unit 16, for example, refers to the relevance information 12 based on the estimated current behavior and identifies the behavior corresponding to the estimated current behavior. The quality judgment unit 16 also identifies all related objects associated with the specific behavior, as well as the degree of relevance between the behavior and each related object, in the relevance information 12.

[0032] Furthermore, the good / bad judgment unit 16 may determine that the current behavior of subject P is good if the object of attention of subject P estimated by the attention estimation unit 15 is included in the identified related object. Conversely, the good / bad judgment unit 16 may determine that the current behavior of subject P is not good if the object of attention of subject P estimated by the attention estimation unit 15 is not included in the related object, and / or if the object of non-attention of subject P is included in the identified related object.

[0033] Alternatively, the good / bad judgment unit 16 can determine the degree of relevance between the subject P's current behavior and the object of attention or non-attention, and may judge the current behavior as ungood if the degree of relevance between the current behavior and the object of attention is lower than a predetermined threshold level, or if the degree of relevance between the current behavior and the object of non-attention is above a predetermined threshold level. Furthermore, the good / bad judgment unit 16 may judge the current behavior as good if the degree of relevance between the current behavior and the object of attention is above a predetermined threshold level, or if the degree of relevance between the current behavior and the object of non-attention is lower than a predetermined threshold level. With this configuration, the goodness or badness of the subject P's current behavior can be judged more appropriately based on the degree of relevance between the subject P's current behavior and the object of attention or non-attention.

[0034] Furthermore, in relational information 12, if there are no related objects corresponding to the object of attention and the object of inattention among the related objects of the behavior corresponding to the current behavior, the object of attention and the object of inattention may be considered to have a degree of relevance to the current behavior of 0.

[0035] Alternatively, the good / bad judgment unit 16 may determine that the current behavior of subject P is unacceptable if, among the related items obtained from the related information 12 for a given current behavior, the related items with a relevance value of a predetermined level or higher are not included in the subject P's attention-focused items or are included in the non-attention-focused items.

[0036] Alternatively, if there are multiple related items obtained from the related information 12 for a given current behavior, the good / bad judgment unit 16 may determine the goodness or badness of the subject P's current behavior based on the percentage of the number of items included in the list of attention targets in the most recent past predetermined time period output by the attention estimation unit 15, relative to the number of related items. For example, the good / bad judgment unit 16 may determine the current behavior to be good when the percentage is above a predetermined threshold, and determine the current behavior to be bad when the percentage is below the threshold.

[0037] Alternatively, if there are multiple related items obtained from the related information 12 for a given current behavior, the good / bad judgment unit 16 may determine whether the subject P's current behavior is good or bad based on the percentage of the sum of the relatedness scores of the items included in the list of attention targets output by the attention estimation unit 15 relative to the sum of the relatedness scores of all related items. For example, the good / bad judgment unit 16 may determine the current behavior is good when the percentage is above a predetermined threshold, and poor when the percentage is below the threshold.

[0038] Furthermore, the current actions that the quality judgment unit 16 considers for judgment may include dangerous work involving danger to the subject P, concentrated work requiring the subject P's concentration, or movement actions involving the subject P's movement. Dangerous work can be defined, for example, as work involving handling tools or objects that could cause injury. Concentrated work can be defined, for example, as work in which the range of the worker's line of sight changes is limited to a predetermined threshold range, or work in which posture changes are monotonous but the speed of movement is faster than a predetermined threshold. With this configuration, the quality of actions that may affect the safety and efficiency of the subject P can be judged objectively.

[0039] The specific actions of dangerous work, concentrated work, and movement that the quality judgment unit 16 makes a judgment on can be predetermined. For example, the relevance information 12 stored in the memory 11 can be associated with each predetermined specific dangerous work, concentrated work, and / or movement, with related objects and degrees of relevance related to each action. The quality judgment unit 16 can make a quality judgment on the current action if the action corresponding to the current action of the subject P estimated by the action estimation unit 14 is included in the relevance information 12, and can refrain from making a quality judgment on the current action if it is not included in the relevance information 12.

[0040] The notification unit 17 notifies subject P when the good / bad judgment unit determines that subject P's current behavior is not good. This allows feedback on the ungood behavior to subject P, who is the one performing it, thereby supporting subject P in efficiently improving their behavior.

[0041] The notification provided by the notification unit 17 may be, for example, an audio message such as "Please pay attention to %%%" from a speaker (not shown) installed in the target space S. Here, "%%%" may be the name of an object of inattention included in the related objects related to the current action, or the name of an object of inattention not included in the object of attention. For example, when the current action is "cutting with a knife," and the related object "fingers of the assisting hand" related to this current action is included in the object of inattention of the subject P, the notification unit 17 may provide a message such as "Please pay attention to the fingers of the assisting hand."

[0042] Alternatively, the notification unit 17 may notify the subject P by, for example, blowing air to the subject P by the fan 5 provided in the target space S, interfering with the subject P by the robot cleaner 6 disposed in the target space S, controlling the lighting of the target space S, or utterance from a device having an audio generation function. This makes it possible to notify the subject P of the result of the quality determination of the action by using various means. The interference with the subject P by the robot cleaner 6 may be, for example, contact of the robot cleaner 6 with the feet of the subject P. Further, the control of the lighting of the target space S may be, for example, change or modulation of the lighting color or illuminance of the lighting fixture L.

[0043] Alternatively, the notification unit 17 may notify the subject P by using signage (or digital signage) that displays text or images on a television screen or a monitor screen. In addition, the notification unit 17 may notify the subject P by stopping images or videos that are played in the background of the working environment of the subject P by a television, a radio, a music player, or the like, or stopping music or muting audio. Alternatively, the above notification may be performed by a special sound from home electric appliances such as a microwave oven, a rice cooker, a washing machine, etc. (such as a sound or voice different from the original operation completion sound of these home electric appliances). Alternatively, the above notification may be performed by outputting a special sound from an intercom disposed in the working room (such as a sound or voice different from the sound notifying the arrival of a visitor), or by utterance from a "talking home appliance" having an audio generation function or a smart speaker.

[0044] [3. Operation of the Action Quality Determination System] Next, the operation of the action quality determination system will be described. FIG. 4 is a flowchart showing the procedure of an action quality determination method executed by the action quality determination system 1. Each step shown in FIG. 4 is executed by the processor 10 of the monitoring device 2, which is a computer constituting the action quality determination system 1. The process shown in FIG. 4 starts when the power of the monitoring device 2 is turned on while the first sensor device 3 and the second sensor device 4 are in an operating state, and is repeatedly executed until the power of the monitoring device 2 is turned off.

[0045] When processing starts, first, the behavior estimation unit 14 of the monitoring device 2 estimates a current behavior, which is the current behavior performed by a target person P, based on sensor information from the first sensor device 3 provided in the target space S, with a person in the target space S set as the target person (S100). Next, the attention estimation unit 15 of the monitoring device 2 detects the line-of-sight direction of the target person P based on sensor information from the second sensor device 4 provided in the target space S, and estimates an attention target, which is a target to which the target person P is directing attention, or a non-attention target, which is a target to which the target person P is not directing attention (S102).

[0046] The quality determination unit 16 of the monitoring device 2 determines whether the current behavior of the target person P is appropriate based on the current behavior of the target person P and the attention target or non-attention target (S104). Then, when the current behavior of the target person P is appropriate (YES in S104), the quality determination unit 16 terminates the present process.

[0047] On the other hand, when the current behavior of the target person P is not appropriate (NO in S104), the notification unit 17 issues a notification to the target person P (S106) and terminates the present process. After the present process is terminated, the behavior estimation unit 14 newly starts processing from step S100.

[0048] In FIG. 4, steps S100, S102, and S104 respectively correspond to the behavior estimation step, the attention estimation step, and the quality determination step in the present disclosure.

[0049] (Other Embodiments) The relevance information 12 may include information that associates a behavior with a non-relevant object that has no relevance to the behavior. When the attention target of the target person P estimated by the attention estimation unit 15 is a non-relevant object associated with the behavior corresponding to the current behavior of the target person P in the relevance information 12, the quality determination unit 16 can determine that the current behavior of the target person P is not appropriate. Accordingly, for example, in the relevance information 12, by associating "urination holding behavior" (e.g., restlessness, etc.) with "any object" as a non-relevant object, the quality determination unit 16 can determine that the urination holding behavior is not appropriate when the target person's attention is directed to some real entity while holding urination.

[0050] In the above-described embodiment, the good / bad judgment unit 16 determines the goodness or badness of the subject P's current behavior by referring to the relevance information 12. Alternatively, the good / bad judgment unit 16 may determine the goodness or badness of the subject P's current behavior using a trained model that has been trained on machine learning to understand the relationship between a person's behavior, an object of attention or inattention, and the goodness or badness of that behavior.

[0051] In the embodiment described above, the behavioral judgment system 1 identifies a person P who is in a target space S, which is a living room within a building, and judges whether the current behavior of that person P is good or bad. Alternatively, the behavioral judgment system 1 may define the target space S as the sensing space (imageable range) of the first and second sensor devices, which are street corner cameras, and identify at least one person in the target space S as the person P, and judge whether the current behavior of the person P (for example, walking while operating a mobile device) is good or bad.

[0052] [4. Effects, etc.] As described above, the behavior quality judgment system 1 determines the quality of the behavior of a person in the target space S, whom the subject P is. The behavior quality judgment system 1 includes a behavior estimation unit 14, an attention estimation unit 15, and a quality judgment unit 16 in the monitoring device 2. The behavior estimation unit 14 estimates the current behavior of the subject P based on sensor information from a first sensor device 3 installed in the target space S. The attention estimation unit 15 detects the direction of the subject P's gaze based on sensor information from a second sensor device 4 installed in the target space S. The attention estimation unit 15 then estimates the attention target, which is the object that the subject P is paying attention to, or the non-attention target, which is the object that the subject P is not paying attention to. The quality judgment unit 16 then determines whether the subject P's current behavior is good or bad based on the subject P's current behavior and the attention target or non-attention target. As a result, the behavioral appropriateness judgment system 1 objectively grasps the subject P's current behavior and attention status to their surroundings based on sensor information from the first sensor device 3 and the second sensor device 4 installed in the target space S, and identifies the subject P's current behavior and whether they are paying attention to or not. Therefore, it can appropriately determine whether the subject P's current behavior is good or not.

[0053] Furthermore, the good / bad judgment unit 16 determines the degree of relevance between the subject P's current behavior and the object of attention or the object of inattention. The good / bad judgment unit 16 then determines that the current behavior is not good if the degree of relevance between the current behavior and the object of attention is lower than a predetermined threshold level, or if the degree of relevance between the current behavior and the object of inattention is above a predetermined threshold level. This allows for a more appropriate determination of whether the subject P's current behavior is good or bad based on the degree of relevance between the subject P's current behavior and the object of attention or the object of inattention.

[0054] Furthermore, the behavior estimation unit 14 recognizes the skeletal movements of the subject P based on sensor information from the first sensor device 3, and can estimate the subject P's current behavior from those skeletal movements. This allows for the estimation of the subject P's current behavior while also considering the subject P's privacy.

[0055] Furthermore, the attention estimation unit 15 recognizes the orientation of the subject P's face based on sensor information from the second sensor device 4, detects the subject P's gaze direction from the face orientation, and estimates whether the object is an object of attention or not. This allows for an objective determination of whether the object of attention or not is the subject P based on the subject P's gaze direction.

[0056] Furthermore, the second sensor device 4 may be a goggle-type or glasses-type eye-tracking sensor. With this, it is possible to easily identify the object of attention or non-attention of the subject P using a goggle-type or glasses-type eye-tracking sensor that is easy to wear during work.

[0057] Furthermore, the current actions that the quality judgment unit 16 considers for judgment may include dangerous work involving the subject P's risk, concentrated work requiring the subject P's concentration, or movement involving the subject P's movement. This allows for an objective judgment of the quality of actions that may affect the safety or efficiency of the subject P.

[0058] Furthermore, the system includes a notification unit that notifies the subject P when the quality judgment unit 16 determines that the current behavior is not good. This allows feedback on the undesirable behavior to be provided to the subject P, who is performing the behavior, thereby supporting the subject P in efficiently improving their behavior.

[0059] Furthermore, the notification unit 17 may also provide notification by blowing air onto the subject P using the fan 5, interfering with the subject P using the robot vacuum cleaner 6, or controlling the lighting of the target space S using the lighting fixture L. This allows the subject P to be informed of the results of the judgment on the appropriateness of the action using various means, not limited to voice.

[0060] Furthermore, the behavioral behavior judgment method executed by the behavioral behavior judgment system 1 includes a behavioral behavior estimation step (S100) in which a person in the target space S is designated as the subject P, and the current behavior is estimated based on sensor information from the first sensor device 3 installed in the target space S. The behavioral behavior judgment method also includes an attention estimation step (S102) in which, based on sensor information from the second sensor device 4 installed in the target space S, the direction of the subject P's gaze is detected, and the attention target (an object that the subject P is paying attention to) or the non-attention target (an object that the subject P is not paying attention to) is estimated. Furthermore, the method for determining whether the behavior is good or bad includes a good or bad determination step (S104) that determines whether the subject P's current behavior is good or bad based on the subject P's current behavior and the object of attention or non-attention. This allows for an objective determination of whether the subject P's current behavior and attention to their surroundings are good or bad, based on sensor information from the first sensor device 3 and the second sensor device 4.

[0061] Since the embodiments described above are for illustrative purposes of the technology described herein, various modifications, substitutions, additions, omissions, etc., can be made within the scope of the claims or equivalents thereof.

[0062] (Note) The above description of embodiments discloses the following technology.

[0063] (Technology 1) An action quality judgment system comprising: an action estimation unit that estimates the current actions of a person in a target space based on sensor information from a first sensor device provided in the target space; an attention estimation unit that detects the direction of the subject's gaze based on sensor information from a second sensor device provided in the target space and estimates the object of attention that the subject is paying attention to or the object of non-attention that the subject is not paying attention to; and a quality judgment unit that determines whether the subject's current actions are good or bad based on the subject's current actions and the object of attention or non-attention. With this configuration, the subject's current actions and attention status to the surroundings are objectively grasped as the subject's current actions and object of attention or non-attention based on sensor information from the first and second sensor devices provided in the target space, so that it is possible to appropriately determine whether the subject's current actions are good or bad.

[0064] (Technology 2) The behavioral behavioral behavioral behavioral behavioral judgment system according to Technology 1, wherein the behavioral behavioral behavioral judgment unit determines the degree of relevance, which is the degree of relationship between the current behavior and the object of attention or the object of non-attention, and determines that the current behavior is not good when the degree of relevance between the current behavior and the object of attention is lower than a predetermined threshold level, or when the degree of relevance between the current behavior and the object of non-attention is equal to or greater than a predetermined threshold level. With this configuration, the behavioral behavioral behavior of the subject can be judged more appropriately based on the degree of relevance between the subject's current behavior and the object of attention or the object of non-attention.

[0065] (Technology 3) The behavior estimation unit recognizes the movement of the subject's skeleton based on sensor information from the first sensor device and estimates the subject's current behavior from the movement of the skeleton, as described in Technology 1 or 2. With this configuration, the subject's behavior can be estimated while also considering the subject's privacy.

[0066] (Technology 4) The attention estimation unit recognizes the orientation of the subject's face based on sensor information from the second sensor device, detects the subject's gaze direction from the orientation of the face, and estimates whether the subject is a subject of attention or a subject of non-attention, as described in any of Techniques 1 to 3. With this configuration, the subject's subjects of attention or non-attention can be objectively determined from the subject's gaze direction.

[0067] (Technical 5) The behavioral goodness judgment system described in Technical 4, wherein the second sensor device is a goggle-type or glasses-type eye-tracking sensor. With this configuration, it is possible to easily determine what the subject is paying attention to or not paying attention to using a goggle-type or glasses-type eye-tracking sensor that is easy to wear during work.

[0068] (Technical 6) The behavioral

[0069] (Technology 7) A behavioral behavioral behavioral behavioral judgment system according to any one of technologies 1 to 6, further comprising a notification unit that notifies the subject when the behavioral behavioral judgment unit determines that the current behavior is not good. With this configuration, ungood behavior can be fed back to the subject who is performing the behavior, thereby supporting the subject in efficiently improving their behavior.

[0070] (Technical 8) The behavioral appropriateness judgment system according to Technical 7, wherein the notification unit provides notification by blowing air onto the subject, interfering with the subject with a robotic vacuum cleaner, controlling the lighting of the subject space, or speaking from a device having a voice generation function. With this configuration, the result of the behavioral appropriateness judgment can be notified to the subject by various means.

[0071] (Technology 9) A method for determining the appropriateness of an action performed by a computer, comprising: an action estimation step in which a person in a target space is the subject, and the current action is estimated based on sensor information from a first sensor device provided in the target space; an attention estimation step in which the direction of the subject's gaze is detected based on sensor information from a second sensor device provided in the target space, and the attention target is estimated to be an object that the subject is paying attention to or an attention target that the subject is not paying attention to; and an appropriateness determination step in which the computer determines whether the subject's current action is appropriate or not based on the subject's current action and the attention target or attention target. This configuration produces the same effects as Technology 1.

[0072] As described above, the behavioral appropriateness assessment system and behavioral appropriateness assessment method related to this disclosure can be used to determine whether a subject's current behavior is appropriate or not.

[0073] 1 Behavior judgment system 2 Monitoring device 3 First sensor device 4 Second sensor device 5 Fan 6 Robot vacuum cleaner 10 Processor 11 Memory 12 Relevance information 13 Program 14 Behavior estimation unit 15 Attention estimation unit 16 Behavior judgment unit 17 Notification unit C1, C2, C3... Outlet L Lighting fixture P Target person S Target space SW Switch

Claims

1. An action quality determination system comprising: an action estimation unit that estimates the current actions of a person in a target space based on sensor information from a first sensor device provided in the target space; an attention estimation unit that detects the direction of the subject's gaze based on sensor information from a second sensor device provided in the target space and estimates the attention target or the non-attention target that the subject is not paying attention to; and a quality determination unit that determines whether the subject's current actions are good or bad based on the subject's current actions and the attention target or non-attention target.

2. The behavioral 3. The behavior estimation unit recognizes the movement of the subject's skeleton based on sensor information from the first sensor device, and estimates the subject's current behavior from the movement of the skeleton, the behavior goodness / badness judgment system according to claim 1.

4. The behavioral goodness / badness judgment system according to claim 1, wherein the attention estimation unit recognizes the orientation of the subject's face based on sensor information from the second sensor device, detects the direction of the subject's gaze from the orientation of the face, and estimates whether the subject is a subject requiring attention or a subject not requiring attention.

5. The behavioral behavior judgment system according to claim 4, wherein the second sensor device is a goggle-type or glasses-type eye-tracking sensor.

6. The behavioral 7. The behavioral 8. The behavioral behavior judgment system according to claim 7, wherein the notification unit provides the notification by blowing air onto the subject, interfering with the subject with a robotic vacuum cleaner, controlling the lighting of the subject space, or uttering from a device having a voice generation function.

9. A computer-based method for determining the appropriateness of an action, comprising: an action estimation step of estimating the current action of a person in a target space based on sensor information from a first sensor device provided in the target space; an attention estimation step of detecting the direction of the subject's gaze based on sensor information from a second sensor device provided in the target space and estimating an object of attention that the subject is paying attention to or an object that the subject is not paying attention to; and an appropriateness determination step of determining whether the subject's current action is appropriate or not based on the subject's current action and the object of attention or the object of non-attention.

Images