Work analysis device, work analysis system, work analysis method, and work analysis program
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- MITSUBISHI ELECTRIC CORP
- Filing Date
- 2025-01-29
- Publication Date
- 2026-08-06
Smart Images
Figure JP2025002798_06082026_PF_FP_ABST
Abstract
Description
Work analysis device, work analysis system, work analysis method, and work analysis program
[0001] This disclosure relates to a work analysis device, a work analysis system, a work analysis method, and a work analysis program.
[0002] There is a proposed apparatus and method for classifying the work in each section using the sections and classes in each time-series sensor data, obtained by measuring the repetitive tasks performed by a worker as the main performer using sensors, determining multiple sensor data sequences (i.e., time-series sensor data) corresponding to the repetitive tasks (i.e., multiple tasks), determining multiple intervals (i.e., segments) obtained by dividing each time-series sensor data in time, and determining a class (i.e., classification) that indicates the type of temporal change in the sensor values contained in each interval, and for classifying the tasks in each interval using the intervals and classes in each time-series sensor data (see, for example, Patent Document 1).
[0003] International Publication No. 2019 / 229943
[0004] However, the conventional technology described above classifies tasks in each section based solely on the worker's actions, which presents a problem in that it is prone to misclassification, where tasks involving similar but different objects (i.e., different tasks) are often categorized as the same task.
[0005] This disclosure is made to solve the problems of the prior art described above, and aims to provide a work analysis device and a work analysis system that can accurately classify the work in each of the multiple sections generated by dividing each of the multiple tasks performed by a work entity, as well as a work analysis method and a work analysis program used in the said work analysis device and work analysis system.
[0006] The work analysis device of this disclosure is characterized by comprising: an action detection unit that acquires a plurality of time-series sensor data corresponding to each of the plurality of tasks performed by a work entity by measuring the plurality of tasks with sensors, and generates a plurality of time-series data of actions including time-series sensor values of actions indicating the actions of the work entity from the plurality of time-series sensor data; an object detection unit that acquires the plurality of time-series sensor data and generates a plurality of time-series data of object types including time-series sensor values of objects indicating the objects handled by the work entity from the plurality of time-series sensor data; and a classification unit that generates a plurality of intervals by temporally dividing each of the plurality of time-series sensor data, determines a time-series data class indicating the type of temporal change shown in the time-series data of actions and the time-series data of object types in each of the plurality of intervals, and generates classification data indicating the classification of each of the intervals in each of the plurality of tasks performed by the work entity.
[0007] The work analysis method of this disclosure is a method performed by a work analysis device and is characterized by comprising the steps of: acquiring a plurality of time-series sensor data corresponding to each of a plurality of tasks, which are generated by measuring a plurality of tasks performed by a work entity with sensors, and generating a plurality of time-series data of actions, which includes time-series sensor values of actions indicating the actions of the work entity, from the plurality of time-series sensor data; acquiring the plurality of time-series sensor data and generating a plurality of time-series data of object types, which includes time-series sensor values of objects indicating the objects handled by the work entity, from the plurality of time-series sensor data; generating a plurality of intervals by temporally dividing each of the plurality of time-series sensor data; determining a time-series data class that indicates the type of temporal change shown in the time-series data of actions and the time-series data of object types in each of the plurality of intervals, and generating classification data that indicates the classification of each of the intervals in each of the plurality of tasks performed by the work entity.
[0008] Using the work analysis device, work analysis system, work analysis method, and work analysis program of this disclosure, it is possible to accurately classify the work in each of the multiple sections generated by dividing each of the multiple tasks performed by a work entity.
[0009] This is a schematic diagram showing a work analysis system according to Embodiment 1. This is a block diagram showing a work analysis device and work analysis system according to Embodiment 1. This is a block diagram showing a modified version of a work analysis device and work analysis system according to Embodiment 1. This is a diagram showing, in tabular form, examples of sensor values for time-series data of motion and sensor values for time-series data of object type used in the work analysis device, along with divided intervals (segments), time steps in each segment, and classification (class). This is a diagram showing examples of time-series data of motion and time-series data of object type used in the work analysis device according to Embodiment 1. This is a diagram showing examples of changes in sensor values for time-series data of motion detected by the motion detection unit of the work analysis device according to Embodiment 1. This is a flowchart showing the operation of the work analysis device according to Embodiment 1. This is a diagram showing an example of the hardware configuration of the work analysis device according to Embodiment 1. This is a block diagram showing the object detection unit of the work analysis device according to a modified version of Embodiment 1. This is a block diagram showing the configuration of the classification unit of the work analysis device according to Embodiment 2. This is a flowchart showing the operation of the classification unit of the work analysis device according to Embodiment 2. This figure shows an example of the generation process for time-series data of operations and time-series data of object types in the work analysis device according to Embodiment 2, and shows an example of the generation process in the probabilistic generation model assumed by the classification unit. This is a block diagram of the work analysis device according to Embodiment 3 (when generating a standard pattern). This is a block diagram of the work analysis device according to Embodiment 3 (when estimating classification data).
[0010] Hereinafter, a work analysis apparatus, a work analysis system, a work analysis method, and a work analysis program according to an embodiment will be described with reference to the drawings. The following embodiments are merely examples, and it is possible to appropriately combine the embodiments and appropriately modify each embodiment. In the figures, components having the same or similar functions are denoted by the same reference numerals.
[0011] <<Embodiment 1>> FIG. 1 is a schematic diagram showing a work analysis system 1 according to Embodiment 1. As shown in FIG. 1, the work analysis system 1 includes a sensor 10, a work analysis apparatus 20, and an output unit 60. The work analysis system 1 is, for example, a part of equipment in a factory or the like. The sensor 10, the work analysis apparatus 20, and the output unit 60 may be mounted on an integrally configured terminal device.
[0012] The sensor 10 is, for example, a depth sensor. The sensor 10 captures a plurality of operations (i.e., u operations) W １ , …, W ｕ performed by a worker 90 as a work subject, and outputs a plurality of time-series sensor data (i.e., u sensor data series) D １ , …, D ｕ corresponding to the plurality of operations W １ , …, D ｕ . The plurality of operations W １ , …, W ｕ are, for example, operations performed repeatedly. The depth sensor includes, for example, a light source that emits infrared rays in a specific pattern and an imaging element that receives the infrared rays reflected by the object, and generates image data (i.e., depth image data) having the depth to the object as pixel values. An imaging device can also be used as the sensor 10. In FIG. 1, the work analysis system 1 has one sensor 10, but may have a plurality of sensors 10. The sensor 10 captures, for example, the operations of the left and right hands 91 of the worker 90 working on the workbench and the objects (e.g., tool 92 or target product 93, etc.) on the workbench.
[0013] The work analysis apparatus 20 performs each operation W of a plurality of operations W １ , …, W ｕ performed by a worker 90 as a work subjectｑ Analyze multiple tasks W １ ..., W ｕ Each task W ｑ Multiple intervals S generated by dividing the space １ ..., each interval S of Sv ｉ The work is classified and classification data (class data column) c ｉ The output is as follows: Here, u is an integer greater than or equal to 2, q is an integer between 1 and u (inclusive), v is a positive integer, and i is an integer between 1 and v (inclusive). The work analysis device 20 is, for example, a computer. This computer may be a computer system that includes multiple information processing devices connected via a network.
[0014] In Figure 1, the work entity is a single worker 90, but the work entity may consist of multiple workers. Furthermore, the work entity may not be a human being, as long as it uses an object to perform a specific task through its moving parts (for example, factory equipment such as a robotic arm, surgical instruments that assist in medical surgery, etc.).
[0015] The output unit 60 is, for example, a display device that displays the results of the work analysis (e.g., classification data or the results of the work quality judgment). The output unit 60 may also include a lamp that lights up to indicate the results of the work analysis, an audio output device that outputs sound to indicate the results of the work analysis, a printing device that prints the results of the work analysis, and so on. If a device that can communicate with the work analysis device 20 (e.g., factory equipment or a smartphone) can be used as an output unit, the work analysis system 1 does not need to have a dedicated output unit 60.
[0016] Figure 2 is a block diagram showing a work analysis device 20 and a work analysis system 1 according to Embodiment 1. As shown in Figure 2, the work analysis device 20 has a motion detection unit 30, an object detection unit 40, and a classification unit 50.
[0017] The motion detection unit 30 detects multiple tasks W performed by the worker 90. １ ..., W ｕ Multiple time-series sensor data D generated by measuring with sensor 10 １ , ..., D ｕObtain multiple time-series sensor data (i.e., u time-series sensor data) D. １ , ..., D ｕ This is a series of repetitive tasks W １ ..., W ｕ The operation detection unit 30 corresponds to multiple time-series sensor data D １ , ..., D ｕ From the time series of sensor values of the worker 90's actions 91, multiple time series data of actions (i.e., u time series data of actions) M １ , ..., M ｕ This generates the operation 91, for example, the hand movements of worker 90. Here, the hand movements of worker 90 include the movements of the worker's fingers.
[0018] The object detection unit 40 detects multiple tasks W performed by the worker 90. １ ..., W ｕ Multiple time-series sensor data D generated by measuring with sensor 10 １ , ..., D ｕ The object detection unit 40 obtains multiple time-series sensor data D. １ , ..., D ｕ From the time series sensor values of the objects handled by worker 90, there are multiple time series data of different object types (i.e., time series data of u objects of different object types). １ , ..., O ｕ The following is generated: The object is, for example, a tool 92 handled by the worker 90, the product 93 being worked on, or both the tool 92 and the product 93. However, the number of objects may be one or three or more.
[0019] The classification unit 50 processes multiple time-series sensor data D １ , ..., D ｕ Each time-series sensor data D ｑ By dividing it in time, multiple intervals (i.e., v segments) S are obtained. １ , ..., S ｖ It generates time-series data M of multiple actions. For example, the classification unit 50 generates time-series data M of multiple actions. １ , ..., M ｕ and time-series data of multiple object types O １ , ..., O ｕBased on this, each time-series sensor data D ｑ Multiple intervals S in １ , ..., S ｖ The classification unit 50 generates multiple intervals S. １ , ..., S ｖ Each section S ｉ In this process, a time-series data class is determined that indicates the type of temporal change shown by the time-series data of the motion and the time-series data of the type of object, and multiple tasks W performed by worker 90 are determined. １ ..., W ｕ Each interval (each segment) S in each of these ｉ Classification data (class data column) c that shows the classification ｉ This generates the following: where v is an integer greater than or equal to 2, and i is an integer between 1 and v (inclusive).
[0020] Figure 3 is a block diagram showing a modified work analysis device 20 and work analysis system 1a according to Embodiment 1. As shown in Figure 3, the work analysis system 1a includes a sensor 10, a sensor data storage device 80, a work analysis device 20, and an output unit 60. The work analysis device 20 receives time-series sensor data D output from the sensor 10 and stored in the sensor data storage device 80. １ , ..., D ｕ The system obtains the following: According to the work analysis system 1a, this is not a work being performed in real time, but rather multiple work W that has been performed in the past. １ ..., W ｕ Classification (class) of each interval (each segment) in each of these: c ｉ It is possible to generate classification data that shows this. In all other respects, the work analysis system 1a in Figure 3 is the same as the work analysis system 1 in Figure 2.
[0021] Figure 4 shows the time-series data M of the operation detected by the operation detection unit 30 of the work analysis device 20 according to Embodiment 1. ｑ The sensor values and time-series data of the type of object detected by the object detection unit 40 O ｑ This figure shows examples of sensor values in tabular format. Figure 4 shows the time-series data of operation M ｑ The sensor values are the x and y coordinates. Figure 4 shows the time-series data of the object type O ｑThe sensor values are numerical values indicating the presence or absence of object #1 (e.g., tool 92) (e.g., 1 for "present" and 0 for "absent"), and numerical values indicating the presence or absence of object #2 (e.g., target product 93) (e.g., 1 for "present" and 0 for "absent").
[0022] Figure 5 shows the time-series data M of the operation used in the work analysis device 20 according to Embodiment 1. ｑ Time-series data of sensor values and object type O ｑ The sensor value and the i-th segment (segment) of the multiple divided segments S ｉ , each segment S ｉ Time step n and classification (class) c in ｉ This figure shows an example in tabular format. Figure 6 shows an example of time-series data of operations and an example of time-series data of object types used in the work analysis device according to Embodiment 1. Figures 5 and 6 show the 1st to 3rd intervals (segments) S in one operation. １ ~S ３ Time series data M in １ ~M ３ And, time-series data of object type O １ ~O ３ and section (segment) S １ ~S ３ The time step n and the classification (class) c in each interval. ｉ An example of classification data is shown.
[0023] Figure 7 shows the time-series data M of the operation detected by the operation detection unit 30 of the work analysis device 20 according to Embodiment 1. ｑ This figure shows an example of changes in sensor values. The time-series data of the operation M ｑ The sensor values are, for example, the distance from sensor 10 to the worker 90's left hand and the distance from sensor 10 to the worker 90's right hand. Time-series data of the operation M ｑ The sensor values may be the coordinates of the worker 90's right hand and the coordinates of the worker 90's left hand. Also, the time series data M of the movement ｑ The sensor values may be the coordinates of the thumb of the worker's hand, or the coordinates of the fingers other than the thumb.
[0024] FIG. 8 is a flowchart showing the operation of the work analysis device 20 according to the first embodiment. FIG. 8 shows a work analysis method performed by the work analysis device 20. First, the work analysis device 20 measures a plurality of operations W １ , …, W ｕ performed by the worker 90 as the work subject with the sensor 10, and generates a plurality of time-series sensor data D １ , …, D ｕ respectively corresponding to the plurality of operations W １ , …, D ｕ . (Step ST1).
[0025] Next, the work analysis device 20 generates a plurality of time-series data M １ , …, M ｕ of a plurality of operations including sensor values of time-series operations showing the operation 91 of the worker 90 from the plurality of time-series sensor data D １ , …, M ｕ . (Step ST2).
[0026] Next, the work analysis device 20 generates a plurality of time-series data O １ , …, O ｕ of types of a plurality of objects including sensor values of time-series objects showing the objects (for example, tool 92, target product 93) handled by the worker 90 from the plurality of time-series sensor data D １ , …, O[[ID=3�]] ｕ . (Step ST3).
[0027] Next, the work analysis device 20 generates a plurality of sections (a plurality of segments) S １ , …, S ｕ by temporally dividing each time-series sensor data D ｑ of the plurality of time-series sensor data D １ [[ID=۴۵]], …, S ｖ . (Step ST4).
[0028] Next, in each section (each segment) S １ , …, S ｖ of the plurality of sections S ｉ , the work analysis device 20 uses the time-series data M ｑ of operations and the time-series data O ｑDetermine the time series data class that indicates the type of temporal change shown, and perform multiple tasks W by worker 90. １ ..., W ｕ Classification data (class data column) c that shows the classification of work in each interval (each segment) in each of the following: ｉ Generate (Step ST5).
[0029] Figure 9 shows an example of the hardware configuration of the work analysis device 20 according to Embodiment 1. As shown in Figure 9, the work analysis device 20 has at least one processor 101 such as a CPU (Central Processing Unit), a memory 102 as a storage device such as RAM (Random Access Memory), a non-volatile storage device 103 such as an HDD (Hard Disk Drive) or SSD (Solid State Drive), and an interface 104 to which output units 60 such as sensors 10 and display devices are connected. These configurations may be made up by a dedicated processing circuit.
[0030] The processor 101 can execute the work analysis method according to Embodiment 1. The work analysis program for executing the work analysis method is recorded on a recording medium (i.e., a storage medium) such as an SD memory card (Secure Digital memory card) or a USB (Universal Serial Bus) memory card, or provided by downloading over a network. The hardware configuration shown in Figure 9 is an example, and the hardware configuration can be modified in various ways.
[0031] As described above, by using the work analysis device 20, work analysis system 1, work analysis method, and work analysis program according to Embodiment 1, the work in each section (each segment) of the multiple sections generated by dividing each of the multiple tasks performed by the worker 90 as the main work subject can be accurately classified based on the worker's actions 91 in the work and the objects used in the work (for example, tools 92, target products 93, etc.).
[0032] Figure 10 is a block diagram showing the object detection unit 40a of the work analysis device 20 according to a modified example of Embodiment 1. As shown in Figure 10, the object detection unit 40a includes a gripping determiner 41 and an object classifier 42. The gripping determiner 41 determines whether the worker's 90 hand is gripping an object. The object classifier 42 classifies the object being handled by the worker's 90 hand. This classification can be performed, for example, by unsupervised learning.
[0033] Furthermore, the gripping determination device 41 determines that the hand is gripping an object when an object that is not the worker's hand is present in the hand area for a predetermined period of time, based on the image of the worker's hand area detected by the sensor 10. The object classifier 42 generates classification data (class data sequence) indicating the classification of the work in each of the multiple segments (each segment) generated by dividing each of the multiple tasks, based on the image of the hand area.
[0034] Furthermore, the gripping determination device 41 may determine gripping based on the distance E between the thumb of the worker's hand and the other fingers of the same hand. For example, the gripping determination device 41 determines that the hand is gripping an object if the distance E is shorter than a predetermined reference value for the gripped position. The object classifier 42 generates classification data (class data columns) that indicate the classification of the work in each of the multiple segments generated by dividing each of the multiple work areas based on the image of the hand area.
[0035] Furthermore, the object detection unit 40 may detect objects detected in the image region where the worker's right hand is located, and objects detected in the image region where the worker's left hand is located.
[0036] Embodiment 2 The work analysis device according to Embodiment 2 differs from the work analysis device 20 according to Embodiment 1 in the configuration of the classification unit. Other than the classification unit, Embodiment 2 is the same as Embodiment 1. Therefore, Embodiment 2 will be described focusing on the differences from Embodiment 1.
[0037] <Configuration of the Classification Unit 50a> Figure 11 is a block diagram showing the configuration of the classification unit 50a of the work analysis device according to Embodiment 2. As shown in Figure 11, the classification unit 50a includes a first classifier 51, a classification data storage device 52, a standard pattern generator 53, a second classifier 54, and a classification data evaluator 55.
[0038] The first classifier 51 processes multiple intervals (i.e., v segments) S １ , ..., S ｖ Each section S ｉ Determine a time series data class that indicates the type of temporal change shown by the time series data of the motion and the time series data of the type of object in the i-th interval (i.e., the i-th interval), and for each interval S ｉ The system generates classification data (class data sequence) C1 as the initial value. The classification data storage device 52 stores the classification data C1 as the initial value. The classification data stored in the classification data storage device 52 is denoted as classification data C.
[0039] The standard pattern generator 53 generates time-series data M of the behavior in classification shown by the classification data. ｑ and time-series data of object types O ｑ Generate a standard pattern, which is a standard pattern.
[0040] The second classifier 54 uses a standard pattern to analyze the time-series data M of the operation. ｑ and time-series data of object types O ｑ The data is classified to generate classification data (class data column) C2.
[0041] The classification data evaluator 55 outputs classification data C2 to the output unit 60 when the classification data stored in the classification data storage device 52 meets predetermined criteria, and updates the classification data C stored in the classification data storage device 52 using the classification data C2.
[0042] <Operation of the Classification Unit 50a> Figure 12 is a flowchart showing the operation of the classification unit 50a of the work analysis device 20 according to Embodiment 2.
[0043] First, the first classifier 51 of the classification unit 50a sorts through multiple intervals S １ , ..., Sｖ Each section S ｉ Determine the time series data class that indicates the type of temporal change shown by the time series data of the motion and the time series data of the type of object in the i-th interval (i.e., the i-th interval), and for each interval S ｉ Classification data C1 is generated as an initial value and stored in the classification data storage device 52 (steps ST11 to ST13).
[0044] The standard pattern generator 53 of the classification unit 50a generates time-series data M of the actions in classification indicated by the classification data. ｑ and time-series data of object types O ｑ A standard pattern is generated (step ST14).
[0045] The second classifier 54 of the classification unit 50a generates classification data C2 to be assigned to each of the multiple intervals generated by dividing the multiple tasks in time, using a standard pattern (step ST15).
[0046] When the classification data evaluator 55 of the classification unit 50a meets a predetermined criterion (YES in step ST16), it updates the classification data (class data sequence) C stored in the classification data storage device with the newly generated classification data C2 and outputs the classification data C2 to the output unit 60 (step ST17).
[0047] If the classification data evaluator 55 of the classification unit 50a does not meet the predetermined criteria (NO in step ST16), it returns to step ST14 without updating the classification data C stored in the classification data storage device 52.
[0048] The standard pattern generator 53 of the classification unit 50a generates a standard pattern as a combination of a probability distribution p(M|c) for calculating the probability that time-series data M of an action is generated from each class (i.e., classification) c, and a probability distribution p(O|c) for calculating the probability that time-series data O of an object type is generated. At this time, the standard pattern generator 53 generates the standard pattern as a Gaussian process for the probability distribution p(M|c) for calculating the probability that time-series data M of an action constituting the standard pattern is generated, and as a multinomial distribution for the probability distribution p(O|c) for calculating the probability that time-series data O of an object type is generated.
[0049] <Generation process of time-series data of motion and time-series data of object type> The classification unit 50a according to Embodiment 2 can achieve classification by assuming a probabilistic generative model. Figure 13 is a diagram showing an example of the generative process in the probabilistic generative model assumed by the classification unit 50a.
[0050] In the generation process assumed by the classification unit 50a, first, the class (i.e., classification) of the immediately preceding interval (i.e., segment) is determined by the following equation (1). ｉ－１ From there, class c of the i-th segment ｉ Let's assume that this is generated.
[0051]
[0052] In the generation process assumed by the classification unit 50a, class c is then determined by the following equation (2). ｉ The time series data x of unit motion from the corresponding Gaussian process ｉ Assume that the following sensor values have been generated.
[0053]
[0054] In the generation process assumed by the classification unit 50a, next, class c is determined by the following equation (3). ｉ The time-series data representing object information is derived from the corresponding multinomial distribution. ｉ Assume that the following sensor values have been generated.
[0055]
[0056] In the generation process assumed by the classification unit 50a, the time-series data x of the operation corresponding to the i-th segment is then generated according to the following equation (4). ｉ By concatenating these, the time-series data M of the operation is obtained. ｑ The following equation (5) generates the time-series data of the object type corresponding to the i-th segment. ｉ By linking them together, the time-series data of the object type O ｑ Let's assume that this is generated.
[0057]
[0058] <Unsupervised Segmentation Method> Unsupervised segmentation is possible by applying the Blocked Gibbs Sampling and Forward Filtering-Backward Sampling methods using the following generation probabilities.
[0059] Class C ｉ From there, the time series data x of the action corresponding to the i-th segment. ｉ And, time-series data of object type o ｉ The probability of generating is given by the following equation (6). Here, p ｍｏｔｉｏｎ (x ｉ | X ｃ ) is the probability of the time series data of the action, and p ｏｂｊｅｃｔ (o ｉ |θ ｃ This represents the probability of a time-series data set for a particular type of object.
[0060]
[0061] In equation (6), the following equations (7a) to (7c) hold true.
[0062]
[0063] Here, DIM is the number of dimensions in the time-series data of the action. Also, Normal() represents the normal distribution. Furthermore, in equation (6), the following equation (8) holds.
[0064]
[0065] Here, Mult() represents the multinomial distribution. In equations (6) to (8) above, i represents the segment number, and c represents the class number. These are the parameters of the standard pattern corresponding to class c.
[0066] These are the mean and variance of the normal distribution followed by the sensor values of the motion generated at the nth time step of the segment corresponding to class c, and can be estimated by Gaussian process regression using the time series data of the motion included in the segment corresponding to class c as input, based on the classification data stored in the classification data storage device 52.
[0067] This is a vector representing the probabilities that each type of object is handled in the segment corresponding to class c. This can be estimated as the frequency with which each type of object appears in the segment corresponding to class c, based on the classification data stored in the classification data storage device 52.
[0068] represents the time step of the segment, and N represents the length of the segment.
[0069] As described above, by using the work analysis device, work analysis system, work analysis method, and work analysis program according to Embodiment 2, the work in each section (each segment) of multiple sections generated by dividing each of the multiple tasks performed by the worker 90 as the main worker can be accurately classified based on the worker's actions 91 and the objects used in the work (e.g., tools 92, target products 93, etc.). Furthermore, since the classification unit 50a optimizes the standard pattern using iteration, improvements in processing speed and classification accuracy can be achieved.
[0070] Embodiment 3 The work analysis device according to Embodiment 3 differs from the work analysis device 20 according to Embodiment 1 in that it has a standard pattern storage device for storing standard patterns and the operation of the classification unit 50b when estimating classification data. The standard patterns are generated by the method described in Embodiment 2. Other than this point, Embodiment 3 is the same as Embodiment 1 or 2. Therefore, Embodiment 3 will be described mainly in terms of the differences from Embodiment 1 or 2.
[0071] Figure 14 is a block diagram showing the work analysis device 20b according to Embodiment 3 (during standard pattern generation). Figure 15 is a block diagram showing the work analysis device 20b according to Embodiment 3 (during classification data estimation). In the work analysis device 20b according to Embodiment 3, the standard pattern storage device 70 stores the standard pattern calculated by the classification unit 50b. During classification data estimation, the classification unit 50b operates as an estimation unit and uses the standard pattern to classify time-series data of operations and time-series data of object types, and generates classification data. The operation of the classification unit 50b is the same as the operation of the second classifier described in Embodiment 2.
[0072] As described above, by using the work analysis device 20b, work analysis system 3, work analysis method, and work analysis program according to Embodiment 3, the work in each section (each segment) of multiple sections generated by dividing each of the multiple tasks performed by the worker 90 as the main work subject can be accurately classified based on the worker's actions 91 in the work and the objects used in the work (e.g., tools 92, target products 93, etc.). Furthermore, the processing speed during estimation can be increased by using standard patterns.
[0073] 1, 1a, 3 Work analysis system, 10 Sensor, 20, 20b Work analysis device, 30 Motion detection unit, 40 Object detection unit, 50, 50a Classification unit, 50b Classification unit (estimation unit), 51 First classifier, 52 Classification data storage device, 53 Standard pattern generator, 54 Second classifier, 55 Classification data evaluator, 60 Output unit, 70 Standard pattern storage device, 80 Sensor data storage device, 90 Worker (work subject), 91 Motion, 92 Object (tool), 93 Object (target product), W ｑ (q = 1, ..., u) Task, D ｑ (q = 1, ..., u) Time-series sensor data (sensor data sequence), M ｑ (q = 1, ..., u) Time-series data of the action (sequence of data for the action), O ｑ (q = 1, ..., u) Time series data of the object (data sequence of the object), S ｉ (i = 1, ..., v) Interval (segment) in each time-series sensor data, u, v are integers greater than or equal to 2, q, i are positive integers.
Claims
1. An action detection unit that acquires multiple time-series sensor data corresponding to each of the multiple tasks performed by a work entity, generated by measuring the multiple tasks with sensors, and generates multiple time-series data of actions, including sensor values of actions in a time series that indicate the actions of the work entity, from the multiple time-series sensor data; an object detection unit that acquires the multiple time-series sensor data and generates multiple time-series data of object types, including sensor values of objects in a time series that indicate the objects handled by the work entity, from the multiple time-series sensor data; and a classification unit that generates multiple intervals by temporally dividing each of the multiple time-series sensor data, determines a time-series data class that indicates the type of temporal change shown by the time-series data of actions and the time-series data of object types in each of the multiple intervals, and generates classification data that indicates the classification of each of the intervals in each of the multiple tasks performed by the work entity.
2. The work analysis apparatus according to claim 1, characterized in that the classification unit generates the plurality of intervals in each time-series sensor data based on the plurality of time-series data of the plurality of actions and the plurality of time-series data of the types of objects.
3. The work analysis apparatus according to claim 1 or 2, characterized in that the operation is the hand movement of the worker as the primary operator of the work.
4. The work analysis apparatus according to any one of claims 1 to 3, characterized in that the object is a tool handled by the work subject, the product to be worked on, or both the tool and the product to be worked on.
5. The work analysis apparatus according to any one of claims 1 to 4, characterized in that the classification unit comprises: a first classifier that generates classification data as initial values based on the classification in each section; a classification data storage device that stores the classification data; a standard pattern generator that generates a standard pattern which is a standard pattern of time-series data of the operation and time-series data of the type of object in the classification indicated by the classification data; a second classifier that uses the standard pattern to classify the time-series data of the operation and time-series data of the type of object and updates the classification data stored in the classification data storage device; and a classification data evaluator that outputs the classification data when the classification data stored in the classification data storage device satisfies predetermined criteria.
6. The work analysis apparatus according to claim 5, characterized in that the standard pattern generator generates the standard pattern by combining a probability distribution p(M|c) for calculating the probability that the time series data of the action is generated and a probability distribution p(O|c) for calculating the probability that the time series data of the type of object is generated, where each of the classification data is denoted as c, the time series data of the action is denoted as M, and the time series data of the type of object is denoted as O.
7. The work analysis apparatus according to claim 6, characterized in that the standard pattern generator generates the probability distribution p(M|c) constituting the standard pattern as a Gaussian process and generates the probability distribution p(O|c) constituting the standard pattern as a multinomial distribution.
8. The work analysis apparatus according to claim 5, characterized in that the classification unit includes an estimation unit that classifies time-series data of the operation and time-series data of the type of object based on the standard pattern and generates the classification data.
9. The work analysis apparatus according to any one of claims 1 to 4, characterized in that the object detection unit includes a gripping determination device that determines whether or not the hand of the person performing the work is gripping the object.
10. The work analysis device according to claim 9, characterized in that the object detection unit determines whether or not the hand of the work subject is grasping the object based on an image of the hand region of the work subject detected by the sensor.
11. The work analysis device according to claim 9, characterized in that the object detection unit determines whether or not the hand of the worker is grasping the object based on the distance between the thumb and other fingers of the worker's hand detected by the sensor.
12. The work analysis apparatus according to any one of claims 1 to 11, characterized in that the object detection unit learns the classification of the object in an unsupervised manner.
13. The work analysis apparatus according to any one of claims 1 to 12, characterized in that the object detection unit detects the object with the right hand and the left hand, respectively.
14. A work analysis system characterized by comprising: a work analysis device according to any one of claims 1 to 13; and the sensor.
15. A work analysis method performed by a work analysis device, comprising: acquiring a plurality of time-series sensor data corresponding to each of a plurality of tasks performed by a work subject by measuring the plurality of tasks with sensors, and generating a plurality of time-series data of actions including time-series sensor values of actions indicating the actions of the work subject from the plurality of time-series sensor data; acquiring the plurality of time-series sensor data and generating a plurality of time-series data of object types including time-series sensor values of objects indicating objects handled by the work subject from the plurality of time-series sensor data; generating a plurality of intervals by temporally dividing each of the plurality of time-series sensor data, determining a time-series data class indicating the type of temporal change shown in the time-series data of actions and the time-series data of object types in each of the plurality of intervals, and generating classification data indicating the classification of each of the intervals in each of the plurality of tasks performed by the work subject.
16. A work analysis program characterized by causing a computer to perform the following steps: acquiring multiple time-series sensor data corresponding to each of the multiple tasks performed by a work entity, generated by measuring the multiple tasks performed by the work entity with sensors, and generating multiple time-series data of actions including time-series sensor values of actions indicating the actions of the work entity from the multiple time-series sensor data; acquiring the multiple time-series sensor data and generating multiple time-series data of object types including time-series sensor values of objects indicating the objects handled by the work entity from the multiple time-series sensor data; generating multiple intervals by temporally dividing each of the multiple time-series sensor data, determining a time-series data class indicating the type of temporal change shown in the time-series data of actions and the time-series data of object types in each of the multiple intervals, and generating classification data indicating the classification of each of the intervals in each of the multiple tasks performed by the work entity.