Information recording system, method, and program

Abstract

An information recording system 100 comprises: cameras 1a, 1b that capture a work area where a worker performs work; measurement sensors 3a to 3p attached to the worker to measure the motion of the worker; and an information recording device 4 that records data, the information recording device 4 including an image analysis unit that analyzes image data captured by the cameras 1a, 1b, a sensor data analysis unit that analyzes sensor data of the measurement sensors 3a to 3p that have measured the motion of the worker, and a data integration unit that, when the image analysis unit was able to detect the worker included in the image data by analysis, integrates an analysis result of the image analysis unit and an analysis result of the sensor data analysis unit and records the integrated result in a database, and that, when the image analysis unit was unable to detect the worker included in the image data by analysis, records the analysis result of the sensor data analysis unit in the database.

Description

Information recording system, method, and program

[0001] The present disclosure relates to an information recording system, method, and program.

[0002] As a method for recording the motion state of a worker, there is posture measurement using a camera. However, since recording is only possible within the field angle of the camera, it is difficult to continuously capture a work environment over a wide range and multiple motions. In addition, when there are obstacles in the work environment or occlusion by other workers, the camera may not be able to accurately record the worker's motions. Accordingly, as a method for overcoming these issues, for example, Patent Document 1 discloses a technology that can record the motion state of a worker by attaching an IMU (Inertial Measurement Unit) sensor to the worker and measuring the movement of the skeleton.

[0003] Japanese Patent No. 7477033

[0004] With the technology disclosed in Patent Document 1, since only the data of the worker's skeletal movement is measured by the IMU sensor, no video recording the worker's motions themselves remains. For this reason, it is not possible to directly tell from the measured data what the worker's motions were performed in relation to, which poses the problems that it is difficult to confirm the detailed content of the motions, and that it is difficult to utilize the data as subsequent training material. Furthermore, when the IMU is used continuously for a long period of time, it accumulates drift errors and bias errors, which may lead to a decrease in measurement accuracy.

[0005] The present disclosure has been made to solve the above-described problems, and an object thereof is to provide an information recording system, method, and program that can improve the accuracy and continuity of data recording.

[0006] To achieve the above-mentioned objectives, the information recording system according to this disclosure comprises: a camera that photographs the work area in which an employee is working; a measurement sensor attached to the employee that measures the employee's movements; and an information recording device that records data, the device including: an image analysis unit that analyzes image data captured by the camera; a sensor data analysis unit that analyzes sensor data from the measurement sensor that measures the employee's movements; and a data integration unit that, if the image analysis unit can detect an employee included in the image data through analysis, integrates the analysis results of the image analysis unit and the analysis results of the sensor data analysis unit and records them in a database, and if the image analysis unit cannot detect an employee included in the image data through analysis, records the analysis results of the sensor data analysis unit in the database.

[0007] According to this disclosure, the information recording system measures the worker's movements using both a camera and a measurement sensor, thereby improving the accuracy and continuity of data recording regarding the worker's movements.

[0008] Figure 3 shows an example of the hardware configuration of the information recording device shown in Figure 3. Figure 4 shows a table of the camera recording database shown in the embodiment. Figure 5 shows a table of the sensor recording database shown in the embodiment. Figure 6 shows a table of the integrated database shown in the embodiment. Figure 7 shows a flowchart of the data recording process shown in the embodiment. Figure 8 shows a flowchart of the camera data recording process shown in the embodiment. Figure 9 shows a flowchart of the sensor data recording process shown in the embodiment.

[0009] Hereinafter, an information recording system 100 including an information recording device 4 according to an embodiment of this disclosure will be described with reference to the drawings. The same or equivalent parts will be denoted by the same reference numerals.

[0010] The information recording device 4 is a device that captures the movements of the worker, who is the target of the movement measurement, with the camera 1, and measures them with the measurement sensor 3 attached to the worker, and generates integrated data by integrating the data captured by the camera 1 and the measurement data measured by the measurement sensor 3. Furthermore, the information recording device 4 is a device that can calibrate the error of the measurement sensor 3 when it detects that the worker captured by the camera 1 is in a predetermined reset position. The information recording system 100 is a system that includes the information recording device 4.

[0011] Figure 1 shows the configuration of the information recording system 100. Figure 2 shows the placement of camera 1 as seen from the ceiling side of the work area. The information recording system 100 is placed in the work area where the worker is performing work and includes camera 1 for photographing the worker, identification marker 2 for identifying the worker, measurement sensor 3 for measuring the worker's movements, and information recording device 4 for recording data acquired from camera 1 and measurement sensor 3. Here, the direction from the floor to the ceiling of the work area is the +Z axis direction, the direction perpendicular to the Z axis direction is the X axis direction, and the direction perpendicular to the X axis direction and the Z axis direction is the Y axis direction. The direction of the arrows on each axis is the + axis direction, and the direction opposite to the arrow is the - axis direction.

[0012] Camera 1 is a camera that photographs workers within the work area and includes, for example, a first camera 1a, a second camera 1b, a third camera 1c, and a fourth camera 1d, all positioned on or near the ceiling of the work area. The first to fourth cameras 1d are positioned at a distance from each other. Camera 1 can be configured using, for example, a camera or video camera that uses an image sensor such as a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor) image sensor. Camera 1 is connected to the information recording device 4 by wired or wireless communication and transmits the captured still images or video data to the information recording device 4.

[0013] The first camera 1a to the fourth camera 1d each have a first shooting range 11a, a second shooting range 11b, a third shooting range 11c, and a fourth shooting range 11d, which are the ranges that fit within their field of view (hereinafter collectively referred to as the shooting range 11). Note that the first shooting range 11a to the fourth shooting range 11d do not overlap. Camera 1 photographs the movements of workers that enter the shooting range 11. For example, if a worker is at position P1 as shown in Figure 2, position P1 is within the first shooting range 11a, so the first camera 1a photographs the worker. If the worker moves from position P1 to position P2, they move out of the first shooting range 11a, so the first camera 1a can no longer photograph the worker. If the worker moves from position P2 to position P3, position P3 is within the second shooting range 11b, so the second camera 1b photographs the worker.

[0014] The workers that camera 1 photographs are those who wear an identification marker 2 somewhere on their body, such as on their head, chest, or back. The identification marker 2 is a unique marker for identifying each worker and functions as a worker ID (Identification). The identification marker 2 can use identification data that can be read from the image captured by camera 1, such as a barcode, worker name, or worker number. The measurement sensor 3 is a sensor attached to the worker's body that can measure the worker's movements. The measurement sensor 3 has a sensor ID, which is unique identification information. The information recording device 4 can detect workers even when camera 1 cannot capture them by linking the sensor ID and the worker ID. The measurement sensor 3 is connected to the information recording device 4 by wired or wireless communication and transmits the measured sensor data to the information recording device 4.

[0015] The measurement sensor 3 can be, for example, an IMU sensor. An IMU sensor is a sensor capable of detecting three-dimensional inertial motion, which is translational and rotational motion in three orthogonal axes. The IMU sensor includes an acceleration sensor and an angular velocity sensor. The acceleration sensor detects translational motion, and the angular velocity sensor detects rotational motion. As a result, the values ​​detected by the acceleration sensor can be used to determine the tilt, velocity, displacement, and translation of the object, which indicate the direction of gravity. The values ​​detected by the angular velocity sensor can be used to determine the object's stopping, fluctuation, rotational motion indicating a constant velocity, and angle with respect to a reference plane. Therefore, by measuring the acceleration and angular velocity of the object to be measured with the IMU sensor, the motion state and posture of the object can be measured.

[0016] The worker is fitted with the following sensors: a first measurement sensor 3a on their head, a second measurement sensor 3b on their right shoulder (assuming the +X axis is to the right), a third measurement sensor 3c on their right elbow, a fourth measurement sensor 3d on their right wrist, a fifth measurement sensor 3e at the base of their right leg, a sixth measurement sensor 3f on their right knee, a seventh measurement sensor 3g on their right ankle, an eighth measurement sensor 3h on their neck, a ninth measurement sensor 3i in the center of their torso, a tenth measurement sensor 3j on their lower abdomen, an eleventh measurement sensor 3k on their left shoulder, a twelfth measurement sensor 3l on their left elbow, a thirteenth measurement sensor 3m on their left wrist, a fourteenth measurement sensor 3n at the base of their left leg, a fifteenth measurement sensor 3o on their left knee, and a sixteenth measurement sensor 3p on their left ankle. By measuring acceleration and angular velocity with each of the first to sixteenth measurement sensors 3a to 3p, the movement state and posture of each part of the worker's body can be determined.

[0017] The information recording device 4 generates and records integrated data by combining the image data received from the camera 1 and the measurement data received from the measurement sensor 3. In addition, the information recording device 4 can calibrate the drift error and bias error of the measurement sensor 3 when it detects that the worker in the image data captured by the camera 1 is in a predetermined reset position. Figure 3 shows the configuration of the information recording device 4. The information recording device 4 includes a communication unit 41 that communicates with the camera 1 and the measurement sensor 3, an operation input unit 42 for inputting instructions from the worker and various data, a notification unit 43 that notifies the worker, a display unit 44 that displays various data, a storage unit 45 that stores various data and programs, and a control unit 46 that executes various controls.

[0018] The communication unit 41 communicates with the camera 1 and the measurement sensor 3, receiving image data of the worker from the camera 1 and sensor data measuring the worker's movements from the measurement sensor 3. The communication unit 41 also transmits drift error correction values ​​and bias error correction values ​​to the measurement sensor 3 for calibrating drift error and bias error. The operation input unit 42 is a device for the worker using the information recording device 4 to input data, instructions, etc. The notification unit 43 is a notification means that notifies the worker, for example, when the worker is asked to assume a reset posture in order to calibrate the error of the measurement sensor 3, or when the reset posture of the worker is detected. The display unit 44 is a device that displays various data, strings, still images, videos, etc. for the worker to see. The display unit 44 also displays content corresponding to the instructions, data, etc. that the worker has entered via the operation input unit 42. The storage unit 45 stores various data used for processing executed by the information recording device 4 and programs for executing various processing described later. Furthermore, the memory unit 45 includes a camera recording database 451 that records the analysis results of images captured by the camera 1 (described later), a sensor recording database 452 that records the analysis results of sensor data from the measurement sensor 3, and an integrated database 453 that integrates the analysis results of images and sensor data.

[0019] The control unit 46 controls various functions performed by the information recording device 4 and includes an image acquisition unit 461 that acquires images taken from the camera 1, an image analysis unit 462 that analyzes the acquired images, a camera data recording unit 463 that records the analysis results, a sensor data acquisition unit 464 that acquires sensor data from the measurement sensor 3, a sensor data analysis unit 465 that analyzes the acquired sensor data, a sensor data recording unit 466 that records the analysis results, a data integration unit 467 that integrates the data, a storage processing unit 468 that stores the data in the storage unit 45, a calibration unit 469 that calibrates the error of the measurement sensor 3, a notification control unit 470 that controls the notification unit 43, and a display control unit 471 that controls the display of the display unit 44.

[0020] The image acquisition unit 461 acquires captured image data from the camera 1 via the communication unit 41. The image data is, for example, a 60 fps (frames per second) video. Each frame of the image data has a corresponding timestamp, and the timestamp increments by one count when one frame is advanced. The image acquisition unit 461 also acquires the position of the camera 1 that took the picture from the camera 1. The position of the camera 1 is a unique coordinate set when it was placed in the work area. The image analysis unit 462 analyzes the image data acquired by the image acquisition unit 461 and detects the worker ID, the worker's posture image, and the reset signal.

[0021] In detail, the image analysis unit 462 detects identification markers 2 within the image of each frame of the video, and uses the identification data read from the identification markers 2 as the worker ID. If identification markers 2 cannot be detected, the image analysis unit 462 determines that there is no worker in that frame and performs identification marker 2 detection on other frames of the video. Furthermore, if identification markers 2 are detected within the image, the image analysis unit 462 estimates the posture and orientation of the worker possessing the identification markers 2 based on the worker's image. The estimation of posture and orientation is performed by obtaining skeletal information using the positions of the worker's joint centers, the centers and endpoints of various body parts, and the centers and endpoints of the eyes and nose as key points in the image, and then estimating based on the skeletal information. Any method, such as deep learning or existing algorithms, may be used to estimate posture and orientation. If there are multiple workers with identification markers 2, the posture and orientation are estimated for each worker.

[0022] Furthermore, the image analysis unit 462 acquires a "True" reset signal if it detects that the worker's posture in the image is one of a predetermined reset posture, such as a T-shape posture with both arms extended horizontally, or an upright posture with arms along the sides of the body. If it does not detect such a posture, it acquires a "False" reset signal. The camera data recording unit 463 records the analysis results of the image analysis unit 462, along with the timestamp of the captured image data and the position of the camera 1 that took the picture, in the camera recording database 451. Details of the camera recording database 451 will be described later.

[0023] The sensor data acquisition unit 464 receives the timestamps of each measurement sensor 3a to 16th measurement sensor 3p, the acceleration and angular velocity data measured by each measurement sensor 3, and the worker ID associated with the sensor ID of each measurement sensor 3 from the measurement sensor 3 via the communication unit 41. The worker ID is registered with the measurement sensor 3 and linked to the sensor ID when the measurement sensor 3 is attached to the worker's body. The sensor data analysis unit 465 analyzes the acceleration and angular velocity data measured by each measurement sensor 3 acquired by the sensor data acquisition unit 464 to determine the worker's position and posture. The sensor data recording unit 466 records the worker's position and posture data determined by the sensor data analysis unit 465, the timestamp at the time of measurement, and the worker ID associated with each measurement sensor 3 in the sensor recording database 452. Details of the sensor recording database 452 will be described later.

[0024] The data integration unit 467 integrates the data contained in the camera recording database 451 recorded by the camera data recording unit 463 and the sensor recording database 452 recorded by the sensor data recording unit 466, and adds them to the end of the integrated database 453 in chronological order of oldest timestamp. Details of the integrated database 453 will be described later. Also, if "True" is recorded in the reset signal item recorded in the camera recording database 451, the data integration unit 467 calculates correction values ​​for drift error and bias error based on the worker's posture and orientation obtained by the image analysis unit 462 and the detected reset posture. The data integration unit 467 records the calculated drift error and bias error correction values ​​in the integrated database 453. The storage processing unit 468 retrieves the camera recording database 451, the sensor recording database 452, and the integrated database 453 from the data integration unit 467 and stores them in the storage unit 45.

[0025] When the calibration unit 469 detects that the integrated database 453 contains "True" as a reset signal, it performs calibration of the measurement sensor 3. Specifically, the calibration unit 469 resets the position information of the measurement sensor 3 to the position information of the camera 1. The calibration unit 469 also uses the drift error and bias error correction values ​​recorded in the integrated database 453 to calibrate the drift error and bias error of the measurement sensor 3. The notification control unit 470 controls the notification unit 43 to notify the operator. The display control unit 471 controls the display of the display unit 44.

[0026] Each of the functions of the control unit 46 described above is implemented by software. In this embodiment, the software that performs the processing to implement each of the functions of the control unit 46 is defined as a data recording processing program. An example of the hardware configuration of the information recording device 4 for executing the data recording processing program is shown in Figure 4.

[0027] The information recording device 4 includes a communication device 501 for communicating with the camera 1 and the measurement sensor 3, an operation input device 502 for the operator to input data, instructions, etc., an alert device 503 for notifying the operator, a display device 504 for displaying various data, a display control 505 for controlling the display of the display device 504, a storage device 506 for storing various programs and various data, a memory 507 for deploying various programs, and a processor 508 for executing various programs. The communication device 501, the operation input device 502, the alert device 503, the display control 505, the storage device 506, the memory 507, and the processor 508 are interconnected via a data bus 509.

[0028] The communication device 501 can communicate with the camera 1 and the measurement sensor 3 and send and receive data. It can be composed of communication devices such as Wi-Fi®, wired or wireless LAN (Local Area Network), Bluetooth®, or infrared communication. The communication device 501 may also be configured using various ports to which devices can be connected, such as a USB (Universal Serial Bus) port or an IEEE 1394 port. The communication device 501 functions as the communication unit 41 shown in Figure 3.

[0029] The operation input device 502 is a device used by the operator to input data, instructions, etc., and can be composed of at least one of the following: a button, a keyboard, a touch panel that works in conjunction with the display unit 44, etc. The operation input device 502 functions as the operation input unit 42 shown in Figure 3. The notification device 503 is a notification means that notifies the operator, and can be composed of the following: a buzzer that emits sound, a speaker, a vibrator that generates vibration, a lamp that emits light, etc. The notification device 503 functions as the notification unit 43 shown in Figure 3.

[0030] The display device 504 is a device that displays various data to be presented to the worker, and can be configured using a display device such as an LCD (Liquid Crystal Display) or an organic EL (Electroluminescence) display. The display device 504 functions as the display unit 44 shown in Figure 3. The display control 505 controls the display of the display device 504. The display control 505 can be configured using a video signal output device such as a video card, GPU (Graphics Processing Unit), or graphics board.

[0031] The storage device 506 is a device for storing various data and programs, and can be configured using a storage device such as an HDD (Hard Disk Drive) or SSD (Solid State Drive). The storage device 506 functions as the storage unit 45 shown in Figure 3. The memory 507 is a device for deploying the various programs stored in the storage device 506. The memory 507 can be configured using a storage element and storage medium such as a RAM (Random Access Memory) or a volatile or non-volatile semiconductor memory such as flash memory. The processor 508 reads the various programs stored in the storage device 506, deploys them in the memory 507, and executes them. The processor 508 can be configured using a processing unit such as a CPU (Central Processing Unit) or an MPU (Micro-processing Unit).

[0032] Next, Figures 5 to 7 show the tables of the camera recording database 451, the sensor recording database 452, and the integrated database 453. Figure 5 shows the table of the camera recording database 451. The camera recording database 451 includes items such as timestamp, the location of camera 1 that photographed the worker, the worker ID of the worker being photographed, the worker's posture image, the worker's posture and orientation, and the reset signal. The data for each item is recorded in association with the timestamp. The timestamp is a timestamp assigned to each frame of the image taken by camera 1. The timestamp is composed of a combination of year, month, day and hour, minute, second, and sequential number, for example, "YYYY / MM / ddThh:mm:ss.000" and "YYYY / MM / ddThh:mm:ss.001" as shown in Figure 5. In the camera recording database 451, the data is sorted in chronological order of the timestamp, from oldest to newest.

[0033] Furthermore, for example, if a worker moves from position P1 to position P2 as shown in Figure 2, the first camera 1a will no longer be able to photograph the worker because the worker will be outside the first possible shooting range 11a. In this case, if the camera 1 is unable to photograph the worker, only the timestamp is recorded, and the other items are either entered with a "-" symbol to indicate that photography was not possible, or left blank.

[0034] The camera position is the position of camera 1 that photographs the worker, and is a coordinate within the work area pre-set for camera 1. For example, when a worker moves from position P1 to position P3 via position P2 as shown in Figure 2, the worker moves from the first shooting range 11a of the first camera 1a to the second shooting range 11b of the second camera 1b. The coordinates indicating the position of the first camera 1a are (I １ ,j １ ) and the coordinates indicating the position of the second camera 1b are (I ２ ,j ２ ) In this case, if the worker is at position P1 within the first shooting range 11a of the first camera 1a, the camera position is (I １ ,j １ ) and when the second camera 1b moves to position P3 within the second shooting range 11b, the camera position becomes (I ２ ,j ２ )

[0035] The worker ID is identification data read from an identification marker 2 attached to the worker's body. The worker ID may be, for example, AAA, BBB, etc. The posture video is image data of a worker within the camera's shooting range 11. The posture and orientation are the worker's posture and orientation obtained by analyzing the posture video with the image analysis unit 462 of the control unit 46. The reset signal is set to "True" if the image analysis unit 462 detects that the worker is in a predetermined reset posture in the posture video it has analyzed, and to "False" if it does not detect this.

[0036] The sensor recording database 452 shown in Figure 6 includes a timestamp, the sensor ID of the measurement sensor 3 attached to the worker being photographed, the worker ID of the worker being photographed, and the worker's position and posture. The timestamp is the date and time when the sensor data measured from the measurement sensor 3 was acquired by the sensor data acquisition unit 464 of the information recording device 4. The timestamp is composed of a combination of year, month, day and hour, minute, second, and sequential number, for example, "YYYY / MM / ddThh:mm:ss.000" and "YYYY / MM / ddThh:mm:ss.001" as shown in Figure 6.

[0037] The sensor ID is unique identification information of the measurement sensor 3 and is acquired from the measurement sensor 3 by the sensor data acquisition unit 464. The sensor ID is, for example, "IAAAA" or "IBBB" as shown in Figure 6. The worker ID is identification information of the worker to whom the measurement sensor 3 is attached. When the measurement sensor 3 is attached, the worker's worker ID is registered and linked to the sensor ID. When the sensor data acquisition unit 464 acquires the sensor ID from the measurement sensor 3, it also acquires the worker ID linked to the sensor ID. The worker ID is, for example, "AAA" or "BBB" as shown in Figure 6. The position and orientation are the position and orientation of the worker at the time of the timestamp. The position and orientation of the worker are determined by the sensor data analysis unit 465 of the information recording device 4 by analyzing the acceleration and angular velocity data measured by the measurement sensor 3 acquired by the sensor data acquisition unit 464.

[0038] Figure 7 shows the table of the integrated database 453. The integrated database 453 includes items such as timestamp, worker ID of the worker being photographed, sensor ID of the measurement sensor 3, posture image, reset signal, worker's position, posture and orientation, camera position, and drift error correction value and bias error correction value of the measurement sensor 3. The integrated database 453 extracts and records each data recorded in the camera recording database 451 and the sensor recording database 452 using the timestamp and worker ID as keys. Of the items in the integrated database 453, the posture image, reset signal, worker's posture and orientation and camera position are extracted from the camera recording database 451, and the sensor ID is extracted from the sensor recording database 452. In addition, if the worker is outside the shooting range 11 of camera 1 and cannot be photographed, the sensor ID, worker's position and posture are extracted from the sensor recording database 452.

[0039] If "True" is recorded in the reset signal item in the camera recording database 451, the data integration unit 467 of the information recording device 4 calculates correction values ​​for drift error and bias error based on the worker's posture and orientation determined by the image analysis unit 462 and the detected reset posture, and records them in the integration database 453. In Figure 7, for example, if the timestamp is "YYYY / MM / ddThh:mm:ss.000", "False" is recorded in the reset signal item. In this case, the data integration unit 467 does not calculate correction values ​​for drift error and bias error. Also, if the timestamp is "YYYY / MM / ddThh:mm:ss.001", "True" is recorded in the reset signal item. In this case, the data integration unit 467 calculates correction values ​​for drift error and bias error. For example, as shown in Figure 7, the drift error correction value is "Nda" and the bias error correction value is "Bna". These correction values ​​are maintained until "True" is recorded in the reset signal field.

[0040] Next, the flow of data recording processing in the present embodiment will be described below with reference to the flowcharts shown in FIGS. 8 to 10. The data recording processing is stored as a data recording processing program in the storage device 506 shown in FIG. 4. The operator activates the information recording device 4 at the timing of recording work within the work area, and inputs a start instruction from the operation input unit 42. For example, the operator selects icons, buttons, and the like displayed on the screen of the display unit 44 of the information recording device 4 via the operation input unit 42. As a result, the processor 508 of the information recording device 4 shown in FIG. 4 reads the data recording processing program stored in the storage device 506 into the memory 507 and executes it.

[0041] The image acquisition unit 461 of the information recording device 4 shown in FIG. 3 executes camera data recording processing (Step S101). The processing flow in the camera data recording processing will be described with reference to the flowchart shown in FIG. 9. The image acquisition unit 461 acquires captured image data from the camera 1 via the communication unit 41 (Step S201). The image data is, for example, a 60 fps (Frames Per Second) moving image, and each frame is provided with a corresponding timestamp. The image analysis unit 462 of the information recording device 4 analyzes the image data (Step S202). Specifically, the image analysis unit 462 first detects the identification marker 2 from the image of each frame of the moving image. The image analysis unit 462 determines whether or not the identification marker 2 has been detected from the image (Step S203). When the identification marker 2 is detected (Step S203; YES), the image analysis unit 462 acquires identification data readable from the identification marker 2 as an operator ID (Step S204).

[0042] The posture and orientation of the worker carrying the identification marker 2 are estimated based on the image of the worker (step S205). The image analysis unit 462 acquires a posture video from which the posture and orientation of the worker can be known (step S206). The image analysis unit 462 acquires a reset signal (step S207). The image analysis unit 462 acquires "True" as the reset signal when it detects that the worker in the image has taken a predetermined reset posture, for example, a T-shaped posture with both arms spread horizontally, an upright posture with arms along the sides of the body, etc., and acquires "False" as the reset signal when no such posture is detected.

[0043] The camera data recording unit 463 of the information recording device 4 records each piece of data acquired by the image analysis unit 462 from step S204 to step S207 in the camera recording database 451, together with the timestamp of the captured image data and the position of the camera 1 that performed the imaging (step S208). Further, in step S203, when the image analysis unit 462 cannot detect the identification marker 2 from within the image (step S203; NO), the camera data recording unit 463 records the timestamp of the captured image data in the camera recording database 451 on the assumption that there is no worker to be imaged within the imaging range 11 of the camera 1 that is the acquisition source of the image data (step S209). The camera data recording unit 463 ends the camera data recording process.

[0044] Here, the description returns to FIG. 8. The sensor data acquisition unit 464 of the information recording device 4 executes a sensor data recording process (step S102). The process flow in the sensor data recording process will be described with reference to the flowchart shown in FIG. 10. The sensor data acquisition unit 464 acquires sensor data measured from the measurement sensor 3 (step S301). Specifically, the sensor data acquisition unit 464 receives, from the measurement sensor 3 via the communication unit 41, the timestamp at the time of measurement by each of the first measurement sensor 3a to the sixteenth measurement sensor 3p, the acceleration and angular velocity data measured by each measurement sensor 3, and the worker ID associated with the sensor ID of each measurement sensor 3.

[0045] The sensor data analysis unit 465 analyzes the acceleration and angular velocity data measured by each measurement sensor 3, acquired by the sensor data acquisition unit 464, to determine the worker's position and posture (step S302). The sensor data recording unit 466 records the worker's position and posture data obtained in step S302, the timestamp at the time of measurement, and the worker ID associated with each measurement sensor 3 in the sensor recording database 452 (step S303). The sensor data recording unit 466 then terminates the sensor data recording process.

[0046] Now, we return to Figure 8. The data integration unit 467 of the information recording device 4 integrates the data contained in the camera recording database 451 recorded in step S101 and the sensor recording database 452 recorded in step S102 (step S103). Specifically, the data integration unit 467 extracts the data recorded in the camera recording database 451 and the sensor recording database 452 using the timestamp and worker ID as keys. Of the items in the integrated database 453, the posture image, reset signal, worker posture and orientation and camera position are extracted from the camera recording database 451, and the sensor ID is extracted from the sensor recording database 452. In addition, if the worker is outside the shooting range 11 of the camera 1 and cannot be photographed, the sensor ID, worker position and posture are extracted from the sensor recording database 452. Furthermore, if "True" is recorded in the reset signal item recorded in the camera recording database 451, the data integration unit 467 of the information recording device 4 calculates correction values ​​for drift error and bias error based on the worker's posture and orientation obtained by the image analysis unit 462 and the detected reset posture. The data integration unit 467 associates each data with a timestamp and records it at the end of the integrated database 453.

[0047] The storage processing unit 468 of the information recording device 4 acquires the camera recording database 451, the sensor recording database 452, and the integrated database 453 from the data integration unit 467 and stores them in the storage unit 45 (step S104). The calibration unit 469 of the information recording device 4 determines whether or not it has detected that the integrated database 453 contains "True" as a reset signal (step S105). If it detects that "True" is contained (step S105; YES), the calibration unit 469 performs calibration of the measurement sensor 3 (step S106). Specifically, the calibration unit 469 resets the position information of the measurement sensor 3 to the position information of the camera 1. The calibration unit 469 also uses the correction values ​​for drift error and bias error recorded in the integrated database 453 to calibrate the drift error and bias error of the measurement sensor 3.

[0048] The data integration unit 467 determines whether or not a termination instruction has been received (step S107). A termination instruction may be given, for example, by an operator via the operation input unit 42, or when termination is necessary due to a malfunction of the information recording device 4, power off, etc. If a termination instruction has been received (step S107; YES), the notification control unit 470 controls the notification unit 43 to notify the operator that a termination instruction has been received (step S108). The notification unit 43 provides notification, for example, by playing a sound through a speaker or flashing a lamp. If there is no termination instruction (step S107; NO), the data integration unit 467 returns to step S101 and executes steps S101 and beyond.

[0049] Furthermore, in step S105, if the calibration unit 469 does not detect that the integrated database 453 contains "True" as a reset signal (step S105; NO), the calibration unit 469 determines whether a predetermined time has elapsed since the previous calibration (step S109). The predetermined time is, for example, the time it takes for the error of the measurement sensor 3 to accumulate to an acceptable value. If a predetermined time has elapsed since the previous calibration (step S109; YES), the calibration unit 469 instructs the notification control unit 470 to control the notification unit 43 to provide notification prompting the operator to assume a reset posture (step S110). For example, the notification unit 43 may provide notification by, for example, playing a message through a speaker, vibrating a device, or flashing a lamp. If a predetermined time has not elapsed since the previous calibration (step S109; NO), the data integration unit 467 returns to step S101 and executes steps S101 and beyond.

[0050] As described above, according to the embodiment, the information recording system 100 measures the worker's movements using both the camera 1 and the measurement sensor, and the information recording device 4 integrates and records the image data captured by the camera 1 and the sensor data measured by the measurement sensor 3. This allows the worker's movement status to be determined from the data measured by the measurement sensor 3, even when the worker cannot be photographed by the camera 1. Therefore, the accuracy and continuity of data recording regarding the worker's movements can be improved.

[0051] Furthermore, the information recording device 4 can calibrate the error of the measurement sensor 3 when it detects that the worker photographed by the camera 1 is in a predetermined reset position. This reduces the errors that become a problem when the measurement sensor 3 is used for a long period of time, thereby enabling long-term recording of a wide range of motion with the measurement sensor 3.

[0052] (Modification 1) In the above embodiment, the camera recording database 451, the sensor recording database 452, and the integrated database 453 are stored in the storage unit 45 of the information recording device 4. However, the invention is not limited to this, and the data may be stored in an external server or storage device that can be connected via the communication unit 41.

[0053] (Modification 2) In the above embodiment, the information recording device 4 includes a communication unit 41, an operation input unit 42, a notification unit 43, a display unit 44, a storage unit 45, and a control unit 46, but is not limited to this. For example, the functions of the control unit 46 may be provided on an external server or other information processing device that can be connected via the communication unit 41, and processed via a network. Alternatively, the notification unit 43, the display unit 44, and the storage unit 45 may each be separate components and connected via a network.

[0054] (Modification 3) In the above embodiment, calibration of the measurement sensor 3 is performed when the integrated database 453 contains "True" as a reset signal, but it is not limited to this. For example, calibration may be performed when it is detected that an operator has moved from the shooting range 11 of camera 1 to the shooting range 11 of another camera 1. Alternatively, calibration may be performed at regular time intervals.

[0055] (Modification 4) In the above embodiment, the information recording device 4 records data to the camera recording database 451, the sensor recording database 452, and the integrated database 453, but it is not limited to this. For example, the recording of data to the camera recording database 451 and the sensor recording database 452 may be performed by an external server or information processing device, and the recorded data from the camera recording database 451 and the sensor recording database 452 may be acquired by the information recording device 4 and used for the integrated database 453.

[0056] (Modification 5) In the above embodiment, the data recording process shown in Figure 8 is performed by executing camera data recording in step S101 and sensor data recording in step S102, but it is not limited to this. The order of execution of camera data recording and sensor data recording may be reversed, or camera data recording and sensor data recording may be executed in parallel. Furthermore, camera data recording and sensor data recording may be performed as a single process.

[0057] Furthermore, in the embodiments of this disclosure, the information recording device 4 can be implemented as a dedicated system. However, it is possible to implement it using a normal computer system instead of a dedicated system. For example, a computer capable of implementing each of the functions of the information recording device 4 described above may be configured by distributing programs for implementing each of the functions of the information recording device 4 on a recording medium such as a computer-readable CD-ROM (Compact Disc Read Only Memory) or DVD-ROM (Digital Versatile Disc Read Only Memory), and installing these programs on the computer. In cases where each function is implemented through a division of labor between the OS (Operating System) and an application, or through cooperation between the OS and an application, only the application may be stored on the recording medium.

[0058] This disclosure allows for various embodiments and modifications without departing from its broad spirit and scope. Furthermore, the embodiments described above are for illustrative purposes only and do not limit the scope of this disclosure. That is, the scope of this disclosure is indicated by the claims, not by the embodiments. Various modifications made within the scope of the claims and the equivalent significance of the disclosure are considered to be within the scope of this disclosure.

[0059] This application is based on Japanese Patent Application No. 2025-067591, filed on 16 April 2025. The entire specification, claims, and drawings of Japanese Patent Application No. 2025-067591 are incorporated herein by reference.

[0060] This disclosure can be suitably used in information recording systems.

[0061] 1 Camera, 1a First camera, 1b Second camera, 1c Third camera, 1d Fourth camera, 2 Identification marker, 3 Measurement sensor, 3a First measurement sensor, 3b Second measurement sensor, 3c Third measurement sensor, 3d Fourth measurement sensor, 3e Fifth measurement sensor, 3f Sixth measurement sensor, 3g Seventh measurement sensor, 3h Eighth measurement sensor, 3i Ninth measurement sensor, 3j Tenth measurement sensor, 3k Eleventh measurement sensor, 3l Twelfth measurement sensor, 3m Thirteenth measurement sensor, 3n Fourteenth measurement sensor, 3o Fifteenth measurement sensor, 3p Sixteenth measurement sensor, 4 Information recording device, 11 Shooting range, 11a First shooting range, 11b Second shooting range, 11c Third shooting range, 11d Fourth shooting range, 41 Communication unit, 42 Operation input unit, 43 Notification unit, 44 Display unit, 45 Storage unit, 46 Control unit, 100 Information recording system, 451 Camera recording database, 452 Sensor recording database, 453 Integrated database, 461 Image acquisition unit, 462 Image analysis unit, 463 Camera data recording unit, 464 Sensor data acquisition unit, 465 Sensor data analysis unit, 466 Sensor data recording unit, 467 Data integration unit, 468 Memory processing unit, 469 Calibration unit, 470 Notification control unit, 471 Display control unit, 501 Communication equipment, 502 Operation input equipment, 503 Notification equipment, 504 Display equipment, 505 Display control, 506 Storage equipment, 507 Memory, 508 Processor, 509 Data bus.

Claims

1. An information recording system comprising: a camera that photographs a work area in which an employee is performing work; a measurement sensor attached to the employee to measure the employee's movements; and an information recording device for recording data, the device comprising: an image analysis unit that analyzes image data captured by the camera; a sensor data analysis unit that analyzes sensor data from the measurement sensor that measures the employee's movements; and a data integration unit that, if the image analysis unit can detect the employee included in the image data through analysis, integrates the analysis results of the image analysis unit and the analysis results of the sensor data analysis unit and records them in a database, and if the image analysis unit cannot detect the employee included in the image data through analysis, records the analysis results of the sensor data analysis unit in the database.

2. The information recording system according to claim 1, wherein the image analysis unit analyzes the image data and, when it detects an identification marker for identifying the worker within the image data, determines the position and posture of the worker.

3. The information recording device further comprises a calibration unit for correcting errors in the measurement sensor, wherein the image analysis unit analyzes the image data and detects that the worker's posture is a predetermined reset posture, the data integration unit determines a correction value to correct the error in the measurement sensor, and the calibration unit corrects the error in the measurement sensor using the correction value determined by the data integration unit, as described in claim 1 or 2.

4. The information recording system according to any one of claims 1 to 3, wherein the measuring sensor is a sensor that measures acceleration and angular velocity data, and the sensor data analysis unit analyzes the acceleration and angular velocity data measured by the measuring sensor to determine the position and posture of the worker.

5. A method to be executed by an information recording system, comprising: analyzing image data of a work area in which an employee is performing work; analyzing sensor data measuring the employee's movements; if the employee is detected in the image data through the analysis, integrating the analysis results of the image data and the analysis results of the sensor data and recording them in a database; and if the employee is not detected in the image data through the analysis, recording the analysis results of the sensor data in the database.

6. A program for causing a computer to perform the following processes: analyzing image data of a work area where a worker is performing work; analyzing sensor data measuring the worker's movements; if the analysis can detect the worker included in the image data, integrating the analysis results of the image data and the analysis results of the sensor data and recording them in a database; and if the analysis cannot detect the worker included in the image data, recording the analysis results of the sensor data in the database.

Images