Work assistance device and work assistance method

Abstract

A work assistance device (5) comprises a processor that executes processing for providing a user with work improvement assistance information, wherein the processor: recognizes, on the basis of a video, one or more work units included in a work process; detects abnormal work in each of the work units; by executing dialog processing based on an abnormal work detection result, acquires improvement assistance information which pertains to each work unit in which the abnormal work has been detected and / or a work process which includes the work unit; and causes a display device to display the improvement assistance information.

Description

Work Support Apparatus and Work Support Method

[0001] The present disclosure relates to a work support apparatus and a work support method that support work improvement based on data obtained from video capturing the state of a worker's work.

[0002] Technologies for supporting work improvement based on data obtained from video capturing a work site at manufacturing and logistics bases have been developed. For example, by extracting so-called bottleneck work among a plurality of work processes and presenting information related to countermeasures therefor, work managers can utilize such information to improve work.

[0003] Conventionally, there is known a work improvement support system that determines whether a target work corresponds to a bottleneck work based on the degree of deviation between the work time required for the work and a target work time for the work, and when it is determined that the target work corresponds to a bottleneck work, reads out work video data related to the work determined to correspond to the bottleneck work from work video data obtained by capturing the state of the work site, and reproduces the work video data (Patent Document 1).

[0004] Japanese Unexamined Patent Application Publication No. 2019-23803

[0005] In the conventional technique disclosed in Patent Document 1, since work video data corresponding to bottleneck work is extracted, a user can examine work improvement points based on the work video data.

[0006] However, in the above-described conventional technique, formulated work improvement measures depend on the skill and proficiency of the user who views the work video data, that is, the person who formulates the measures, so it cannot be expected that appropriate improvement measures will be stably formulated.

[0007] Furthermore, it is desirable that information for supporting work improvement is provided not only for the entire work but also for more subdivided work units, but the above-described conventional technique does not give any consideration to such a demand.

[0008] Therefore, the main purpose of this disclosure is to provide a work support device and a work support method that can generate information useful for improving work based on data obtained from video footage of workers performing their tasks.

[0009] The work support device of this disclosure is a work support device that supports the improvement of work based on data obtained from video footage of a worker's work, and comprises a processor that performs processing to provide the user with work improvement support information, wherein the processor recognizes one or more work units included in a work process based on the video footage, detects abnormal work in each of the work units, and performs dialogue processing based on the detection result of the abnormal work, thereby acquiring the improvement support information relating to each of the work units in which the abnormal work was detected and at least one of the work process including the work units, and displays the improvement support information on a display device.

[0010] Furthermore, the work support method of this disclosure is a work support method that performs a process to provide the user with work improvement support information based on data obtained from video footage of a worker's work, wherein the computer recognizes one or more work units included in the work process based on the video footage, detects abnormal work in each of the work units, and performs dialogue processing based on the detection results of the abnormal work, thereby acquiring the improvement support information relating to each of the work units in which the abnormal work was detected and at least one of the work process including said work units, and displays the improvement support information on a display device.

[0011] According to this disclosure, it is possible to generate information useful for improving work processes based on data obtained from video footage of workers performing their tasks.

[0012] Figure 1 shows an overall configuration diagram of the information processing system according to this embodiment. Figure 1 shows an example of multiple work processes to be analyzed by the work process analysis server. Figure 1 shows a hardware configuration diagram of the work process analysis server. Figure 1 shows a functional block diagram of the work process analysis server. Figure 1 shows an example of work log data. Figure 1 shows an example of shipment count data. Figure 1 shows an example of work breakdown data (before data cleansing). Figure 2 shows an example of prompts used for agent selection and question prompt creation. Figure 3 shows an example of a first prompt for factor analysis and improvement proposals. Figure 4 shows an example of a second prompt for factor analysis and improvement proposals. Figure 5 shows an explanation of the data cleansing process for work breakdown data. Figure 6 shows an example of work breakdown data (after data cleansing). Figure 7 shows an example of a prompt for summary generation. Figure 8 shows an example of a period setting screen. Figure 9 shows an example of the top page of the improvement support screen. Figure 1 shows an example of a detail page of the improvement support screen. Figure 1 shows an example of dashboard display data. Figure 1 shows a flowchart illustrating the process of generating work breakdown data by the work process analysis server. Figure 1 shows a flowchart illustrating the process of generating a summary by the work process analysis server. Figure 1 shows a flowchart illustrating the process of transmitting data by the work process analysis server.

[0013] The first invention made to solve the aforementioned problems is a work support device that supports the improvement of work based on data obtained from video footage of an worker's work, comprising a processor that performs processing to provide the user with work improvement support information, wherein the processor recognizes one or more work units included in a work process based on the video footage, detects abnormal work in each work unit, and performs dialogue processing based on the detection result of the abnormal work, thereby acquiring the improvement support information relating to each work unit in which the abnormal work was detected and at least one of the work process including said work unit, and displays the improvement support information on a display device.

[0014] According to this, it becomes possible to generate improvement support information useful for improving work processes based on data obtained from video footage of workers performing their tasks.

[0015] Furthermore, the second invention is configured such that in the dialogue processing, the detection result of the abnormal operation is input to the generation model, and the improvement support information is acquired based on the output of the generation model.

[0016] According to this, improvement support information can be obtained by performing a predetermined dialogue process.

[0017] Furthermore, the third invention is configured such that the processor acquires improvement proposals for each work unit as improvement support information.

[0018] According to this, users can appropriately consider areas for improvement in their work based on improvement proposals for more granular work units.

[0019] Furthermore, the fourth invention is configured such that the processor acquires improvement proposals for the work process as improvement support information.

[0020] According to this, users can more appropriately consider areas for improvement in their work based not only on improvement proposals for each individual work unit, but also on improvement proposals for the work process that includes those work units.

[0021] Furthermore, the fifth invention is configured such that the processor acquires summaries of proposed improvements to a plurality of work units as proposed improvements to the work process.

[0022] According to this method, by summarizing improvement proposals for work related to multiple work units, it becomes possible to appropriately obtain improvement proposals for work processes that include those work units.

[0023] Furthermore, the sixth invention is configured such that the processor acquires a summary of proposed improvements to the work process corresponding to the work process performed during a predetermined period, and generates a dashboard screen that displays the proposed improvements to the work process.

[0024] According to this, users can easily review proposed improvements to work processes for a desired period (i.e., a predetermined period set in advance).

[0025] Furthermore, the seventh invention is configured such that the processor generates a dashboard screen that displays information about the plurality of work units in which the abnormal work was detected, and the improvement support information for each of the work units.

[0026] According to this, users can easily check improvement support information for multiple work units in which abnormal work was detected.

[0027] Furthermore, the eighth invention is configured such that the processor displays the corresponding improvement support information on the dashboard screen based on a user's operation on any of the information relating to the plurality of work units.

[0028] According to this, users can easily select a desired work unit from information on multiple work units in which abnormal work was detected and view information to support its improvement.

[0029] Furthermore, the ninth invention is a work support method that performs a process to provide a user with information to support the improvement of the work based on data obtained from video footage of an worker's work, wherein a computer recognizes one or more work units included in a work process based on the video footage, detects abnormal work in each of the work units, and performs a dialogue process based on the detection result of the abnormal work, thereby acquiring the improvement support information relating to at least one of the work units in which the abnormal work was detected and the work process including the work units, and displays the improvement support information on a display device.

[0030] According to this, it becomes possible to generate improvement support information useful for improving work processes based on data obtained from video footage of workers performing their tasks.

[0031] The embodiments of this disclosure will be described below with reference to the drawings.

[0032] The information processing system 1 shown in Figure 1 performs process analysis processing related to the work process based on data obtained from video footage of a worker's work. The information processing system 1 includes a camera 2, a recorder 3, a management system 4, a work process analysis server (an example of a work support device) 5, an artificial intelligence (AI) server 7, and a user terminal 8. These components of the information processing system 1 can communicate with each other via a known communication network 11, such as the Internet or a dedicated line network, as needed.

[0033] Camera 2 is an omnidirectional camera or box camera that is placed in a workplace such as a manufacturing or logistics base to film workers performing their tasks and generate video footage of the work. Although only one camera 2 is shown in Figure 1 for convenience, typically, one or more cameras 2 may be installed in the information processing system 1 to correspond to each work process (see Figure 2).

[0034] Recorder 3 records the video output from camera 2. However, in the information processing system 1, recorder 3 is not essential, and the video generated by camera 2 may be stored in camera 2 itself or in other recording devices.

[0035] Management system 4 includes information processing devices such as PCs and servers that execute processes to manage the work of multiple workers at the target facility (for example, manufacturing equipment or a logistics warehouse). When management system 4 manages work in a logistics warehouse, for example, it can implement the functions of a Warehouse Management System (WMS).

[0036] Furthermore, the management system 4 generates work logs 12 (i.e., data obtained at the work site) as information indicating the work status (or work results) of the target equipment, and sequentially stores them in a storage device such as a storage device. The work logs 12 include, for example, the start time and end time of each work process. The work logs 12 may also include information on the operating status of the hardware at the work site, information on the equipment such as communication history, etc.

[0037] The work process analysis server (hereinafter referred to as the "analysis server") 5 acquires video footage related to each work process from the camera 2 or recorder 3, and performs analysis on each work process based on that video footage.

[0038] The analysis results from analysis server 5 can also be used for simulations related to the target work process. For example, it would be desirable to use the analysis results to perform simulations related to work time and work costs (e.g., labor costs).

[0039] The generative AI server 7 performs dialogue processing using a known generative model constructed by deep learning with a large dataset. Known large-scale language models (LLMs) and large-scale vision language models (VLMs) are used as generative models. This enables natural language analysis and generation, as well as integrated analysis and generation of images and language. In this embodiment, the generative AI server 7 performs processing such as analyzing the causes of abnormal work and proposing improvements related to abnormal work, in response to requests from the analysis server 5.

[0040] User terminal 8 (an example of a display device) is used by users such as the work manager or the worker themselves, and consists of a PC, tablet, and smartphone. User terminal 8 is used for various settings, input of operation instructions, and viewing of information in the information processing system 1. User terminal 8 may be set up for multiple users.

[0041] Furthermore, in the information processing system 1, at least some of the camera 2, recorder 3, management system 4, generation AI server 7, and user terminal 8 are not essential components of the information processing system 1. In other words, at least some of them may be used as external devices or systems that are not included in the information processing system 1 (i.e., they cooperate with the information processing system 1).

[0042] FIG. 2 shows a plurality of types of work processes in a distribution warehouse as an example of objects to be analyzed by the analysis server 5. The plurality of types of work processes include: (A) inspection of products received from a manufacturing factory or the like, (B) product picking, (C) product packaging, (D) packing of products to be shipped, and (E) shipment of products to a customer or the like.

[0043] In each of the work processes (A) to (E), the same work by each worker is repeatedly performed on sequentially generated work target articles (here, products). The work status of each worker W is respectively photographed by a correspondingly installed camera 2.

[0044] Next, the hardware configuration of the analysis server 5 will be described with reference to FIG. 3.

[0045] The analysis server 5 can be configured from a computer having a known hardware configuration. For example, the analysis server 5 includes a processor 15, a memory 16, a storage 17, and a network I / F (interface) 18.

[0046] The processor 15 includes a CPU, a GPU, etc., and executes processing for analyzing each work process based on a predetermined control program. The memory 16 includes a RAM used as a temporary data storage area. The storage 17 includes an SSD, an HDD, or the like for storing data for a long period of time. The network I / F 18 is connected to the communication network 11, and performs transmission and reception of data with other devices and establishment of communication.

[0047] Furthermore, the functions of the analysis server 5 (an example of a computer) may be implemented by a distributed processing system or a virtual server. In that case, the analysis server 5 can be configured by a distributed system including servers, data centers, storages, network devices, and the like that are respectively connected via a communication network. Note that in the present disclosure, the term "apparatus" is not limited to a single device, but is a concept including a configuration in which a plurality of devices function in cooperation with each other.

[0048] Furthermore, except for the difference in functions, the generative AI server 7 can also employ the same configuration as the analysis server 5 described above. Note that the analysis server 5 may also have the functions of the generative AI server 7 (that is, the analysis server 5 may execute interactive processing using a generative model).

[0049] Next, the functions of the analysis server 5 will be described with reference to FIG. 4.

[0050] The analysis server 5 has the following functions: log data acquisition 21, video acquisition 22, work decomposition 23, abnormality detection 24, factor analysis and improvement proposal 25, work process DB (database) 26, data cleansing 27, summary generation 28, and dashboard display 29. At least some of these functions are implemented by the processor 15 executing a control program in the configuration of the analysis server 5 as described above.

[0051] In log data acquisition 21, data of the work log 12 is acquired from the management system 4. For example, the analysis server 5 can acquire the accumulated data of the work log 12 at the timing when the data of the work log 12 for a predetermined period (for example, one day) is accumulated in the management system 4. The acquired data of the work log 12 is stored in the work process DB 26 on the storage 17.

[0052] As shown in FIG. 5, for example, the data of the work log 12 includes a management ID, a cycle start time, a cycle end time, and the like. Furthermore, as shown in FIG. 6, for example, the data of the work log 12 may also include the number of goods shipments per day.

[0053] The management ID is information for identifying one target work process (for example, one picking operation shown in FIG. 2(B)). The cycle start time and the cycle end time are respectively the start time and end time of one target work process.

[0054] In video acquisition 22, videos related to each work process are sequentially acquired from the camera 2 or the recorder 3. The acquired video data is stored in the work process DB 26.

[0055] In task decomposition 23, processing is performed using the work log and video related to the work process stored in the work process DB 26. First, in task decomposition 23, one or more cyclical tasks (an example of a work unit) that are repeated in each work process are recognized based on the video related to the work process. In other words, the analysis server 5 can decompose (i.e., subdivide) each work process into one or more work units. At this time, the analysis server 5 can detect cyclical tasks by performing interaction recognition processing on the video. The analysis server 5 can also obtain the time required for each cyclical task.

[0056] For example, if the work process being processed is a picking operation (see Figure 2(B)), multiple cycle operations may be recognized, such as moving the picking cart to the target shelf, picking up the items stored on the shelf, and moving the picking cart loaded with items to the location of the next process, the packaging operation. However, depending on the type of work process, only one cycle operation may be recognized from the work process.

[0057] Furthermore, in the work breakdown 23, analysis result data (hereinafter referred to as "work breakdown data") is generated based on the data related to each recognized cycle task and the corresponding work log data. The generated work breakdown data is stored in the work process DB 26.

[0058] The work breakdown data includes, for example, the task type, task number, number of cycle classifications, and cycle time, as shown in Figure 7.

[0059] The task type is information about the type of work process (for example, identification information for one of the work processes (A)-(E) shown in Figure 2). The task number is the sequential number of the target work process in a series of similar work processes that are repeatedly performed. The cycle classification number is the number of cycle tasks included in each work process. The cycle time is the time taken for each cycle task. For example, if the number of cycle tasks is 3, the cycle time includes three times corresponding to those three tasks (for example, 24 seconds, 36 seconds, and 57 seconds). The cycle classification number and cycle time are data obtained from video related to the work process (an example of work recognition data).

[0060] In the work breakdown data shown in Figure 7, the management ID, cycle start time, and cycle end time are obtained from the work log 12 described above. In other words, the data generated by the work breakdown 23 is associated with the data from the work log 12 related to the same work process (in this case, the management ID, cycle start time, and cycle end time). The presence or absence of an anomaly and the anomaly classification are information added by the anomaly detection 24, which will be explained next. Furthermore, the exclusion flag is information added by the data cleansing 27, which will be described later.

[0061] In the anomaly detection 24, abnormal work in the target work process is detected based on the work breakdown data obtained by the work breakdown 23 (data excluding the presence or absence of anomalies, anomaly classification, and exclusion flag shown in Figure 7). Abnormal work includes, for example, work in which the time taken for each work unit falls outside the acceptable range. For example, the analysis server 5 can detect cycle work in which the time taken falls outside the acceptable range based on the time data contained in the video data.

[0062] Furthermore, in the anomaly detection 24, based on video footage related to the work process, tasks performed using incorrect work procedures or tasks that deviate from safety standards may be detected as abnormal tasks. The detected abnormal tasks are associated with one of several pre-set anomaly classifications (i.e., classified). For example, the analysis server 5 can detect cyclical tasks performed using incorrect work procedures or cyclical tasks that deviate from safety standards by executing known behavior recognition processing on the video footage.

[0063] The analysis server 5 can detect abnormal operations for each cycle operation using the anomaly detection 24 and classify them.

[0064] The data obtained by the anomaly detection 24 (in this case, the presence or absence of an anomaly and the anomaly classification) is added to the work breakdown data in the work process DB 26. In other words, the work breakdown data generated by the work breakdown 23 described above is updated based on the results of the anomaly detection 24.

[0065] In the work breakdown data shown in Figure 7, the presence or absence of an abnormality indicates whether there is an abnormal operation in the target work process ("1" or "0"). The abnormality classification is identification information related to the classification of the detected abnormal operation.

[0066] In Factor Analysis and Improvement Proposal 25, based on the detection results of abnormal work, the cause of the abnormality is analyzed, and improvement proposals for work to prevent the abnormality from occurring are generated.

[0067] The analysis server 5 can send (i.e., input) the detection result of the abnormal work and a prompt instructing the generation AI server 7 to analyze its cause in order to perform an analysis of the cause of the anomaly. At this time, the analysis server 5 can send the video of the corresponding work along with the detection result of the abnormal work. In response, the generation AI server 7 generates the result of the cause analysis (text) and sends (i.e., outputs) it to the analysis server 5.

[0068] Furthermore, the analysis server 5 can send a prompt to the generation AI server 7 instructing it to propose improvements to the work based on the results of its factor analysis, in order to generate improvement plans. In response, the generation AI server 7 generates improvement plans for the abnormal work and sends them to the analysis server 5. Note that the instruction to the generation AI server 7 to propose improvements to the work may be sent together with the instruction for the factor analysis described above.

[0069] The analysis server 5 can perform a factor analysis of abnormal work and generate improvement proposals using the factor analysis and improvement proposal 25, for example, as follows:

[0070] First, the analysis server 5 instructs the generation AI server 7 to select appropriate agents (i.e., experts). The analysis server 5 also instructs the generation AI server 7 to create prompts that include questions for those agents. At this time, the analysis server 5 can send prompts to the generation AI server 7 instructing them to select agents and create question prompts, for example, as shown in Figure 8.

[0071] Next, the analysis server 5 instructs each agent selected by the generation AI server 7 to perform an analysis of the cause of the abnormal operation. At this time, the analysis server 5 can send prompts to the generation AI server 7 to each agent (in this case, logistics management specialists) asking questions about the issues with the operation, as shown in Figure 9, for example.

[0072] Subsequently, the analysis server 5 instructs the generation AI server 7 to set up an organizer (response summarizer) to consolidate the responses from the multiple agents. Furthermore, based on the responses from the multiple agents, the analysis server 5 instructs the organizer to respond with the issues of the cycle work (i.e., the causes of abnormal work) and proposed improvements. At this time, the analysis server 5 can send a prompt to the generation AI server 7 requesting the organizer to provide the issues of the work and proposed improvements, for example, as shown in Figure 10.

[0073] This allows the analysis server 5 to obtain analysis of the causes of abnormal work and suggestions for improving work from the generation AI server 7.

[0074] Although not illustrated here, the data obtained from the factor analysis and improvement proposal 25 is added to the work decomposition data in the work process DB 26 as data related to the corresponding work process. Note that in the factor analysis and improvement proposal 25, the improvement proposal may be omitted and only the factor analysis may be performed.

[0075] In data cleansing 27, a data cleansing process is performed on the work breakdown data in the work process DB 26. In the data cleansing process, data unsuitable for simulation is removed from the work breakdown data. Here, in order to improve the accuracy of simulations of future work results, it is necessary to retain abnormal data within the appropriate range as simulation data. However, accidental abnormal data outside the appropriate range should be excluded as unsuitable data because retaining it would lead to a deterioration in simulation accuracy. In other words, useful simulation results can be obtained by repeatedly adjusting the setting of the appropriate range and excluding unsuitable data.

[0076] For example, inappropriate data may include data with cycle times above a threshold or data with cycle times below a threshold (see the data labeled (A) and (B) in Figure 7, respectively). Inappropriate data may also include data with cycle times outside the appropriate range (for example, values ​​that should not exist). In addition, inappropriate data may include data where the results of sensing related to the work have not been detected (for example, data where the video could not be acquired properly) (see the data labeled (C) in Figure 7). In addition, inappropriate data may also include data where the work log 12 has not been detected (for example, data where at least a portion of the work log 12 could not be acquired properly).

[0077] The work decomposition data after data cleansing is given an exclusion flag to identify data to be excluded (see the data labeled (D) in Figure 11), as shown in Figure 11. The exclusion flag indicates whether the corresponding data should be excluded from the data used in the simulation ("1") or not ("0").

[0078] Each work process that makes up the work breakdown data described above includes data based on the work log 12. However, there are work processes for which a work log 12 is not generated even under normal circumstances. In this case, the data cleansing 27 can supplement the data for work processes for which work logs cannot be obtained with data obtained from video footage related to the work process.

[0079] The supplemented data is added as data relating to one work process that constitutes the work breakdown data, for example, as shown by the symbol (E) in Figure 11. In the data labeled (E) in Figure 11, the cycle start time and cycle end time, which cannot be obtained from the work log 12, are supplemented from the video related to the work process. However, this process of supplementing work log data from video may be omitted.

[0080] Figure 4 shows an example in which the analysis server 5 has a work process DB 26, and processing by data cleansing 27 is performed in that database. On the other hand, the functions of the work process DB 26 described above may be implemented by a database server provided separately from the analysis server 5.

[0081] Through the data cleansing process, corrected work breakdown data (hereinafter referred to as "corrected data") is generated, for example, as shown in Figure 12. In the corrected data shown in Figure 12, data related to the work process in which abnormal work occurred is supplemented with data obtained from the factor analysis and improvement proposal 25 (in this case, improvement proposals).

[0082] In summary generation 28, a process (hereinafter referred to as "summary generation process") is executed to generate a summary of proposed improvements to work processes performed during a predetermined period (hereinafter referred to as the "summary period").

[0083] The analysis server 5 sends a prompt to the generation AI server 7 instructing it to perform a factor analysis of the abnormal work during the summarization period and to generate a summary of improvement proposals, in order to generate a summary of improvement proposals. At this time, the analysis server 5 can send a prompt to the generation AI server 7 instructing it to summarize the improvement proposals for the work, for example, as shown in Figure 13. When instructing the generation AI server 7 to generate a summary of improvement proposals, the analysis server 5 may also send improvement proposals for multiple cycle work as described above, in addition to the factor analysis of the abnormal work.

[0084] The dashboard display 29 shows information including the factor analysis and improvement suggestions 25 mentioned above, and the processing results from the summary generation 28 (hereinafter referred to as "improvement support information"). For example, a user can access the analysis server 5 using the user terminal 8 to display an improvement support screen (an example of a dashboard screen) containing improvement support information on the display.

[0085] In this case, if the user wants to display an improvement support screen on the display that includes a summary of improvement suggestions for the work during the summary period as improvement support information, they can set a desired period. For example, the user can perform settings related to the generation of improvement support information, including the summary period, on the period setting screen 41 shown in Figure 14.

[0086] On the period setting screen 41, the summarization period, target time, and level of detail of the summary are set. The target time is the time during which the data used to generate the summary of improvement proposals is acquired. The level of detail of the summary is the target value (or upper limit) of the number of words in the summary (text) of the improvement proposals. Here, the target value of the number of words in the summary is set to 500 words.

[0087] The dashboard display 29 shows the top page 51 of the improvement support screen on the user terminal 8, for example, as shown in Figure 15. The top page 51 includes a work selection field 52 and an improvement support information field 53. In the work selection field 52, multiple types of work processes (see Figure 2) are displayed for the user to select. In the improvement support information field 53, improvement support information corresponding to the work process selected by the user is displayed.

[0088] Here, an example is shown where the user has selected "Incoming Inspection" (corresponding to (A) in Figure 2) in the task selection field 52.

[0089] As a result, the improvement support information field 53 displays a video (or image) 55 of the selected work process (in this case, receiving inspection) and text 56 indicating the cause analysis of the abnormal work related to the work process and suggestions for improving the work.

[0090] Here, an example is shown where the summary period is set to today. However, the user can change the summary period, etc., by pressing the period setting button 57, which displays the period setting screen 41 described above.

[0091] Furthermore, when a user presses the details confirmation button 58 on the top page 51, the details page 61 is displayed, for example, as shown in Figure 16. The details page 61 includes a cycle work display field 62 that shows information about each cycle work (here, No. 1-6) related to the work process selected on the top page 51. The cycle work display field 62 displays a graph (here, a bar graph) showing the work time and non-work time for each cycle work. In addition, the graph indicates whether or not there is an abnormality in each cycle work by the presence or absence of a shape indicating "abnormality detected".

[0092] In the cycle work display field 62, when the user selects a graph for a desired cycle work, the video display field 63 displays a video related to the selected cycle work.

[0093] Furthermore, the anomaly detection result display field 64 displays, as text, information regarding the presence or absence of anomalies related to the cycle work selected by the user (see "Anomaly Detection Results"), the issues (i.e., the results of the factor analysis), and improvement suggestions.

[0094] Furthermore, the dashboard display 29 allows the user terminal 8 to display data related to the analysis results from the analysis server 5 (hereinafter referred to as "dashboard display data"), for example, as shown in Figure 17.

[0095] The dashboard display data includes task type, number of tasks, start date, end date, standard time, anomaly rate, anomaly duration, anomaly classification, and anomaly rate for each classification.

[0096] The task count is the number of times each type of work process was executed during the set summary period. The start date and end date are the dates indicating the beginning and end of the summary period. Standard time is the standard value of the time required for each type of work process executed during the summary period. The anomaly rate is the ratio of the number of work processes in which anomalies were detected to the total number of work processes performed during the summary period for a given type of work process. Anomaly duration is the sum of the time in which anomalies were detected in each work process performed during the summary period for a given type of work process. Anomaly classification is the identification information for all anomalies detected in each work process performed during the summary period for a given type of work process. The anomaly rate for each classification is the anomaly rate for that specific anomaly classification.

[0097] Next, with reference to Figure 18, the process of generating work breakdown data by the analysis server 5 will be explained.

[0098] First, the analysis server 5 acquires video footage related to the work process to be analyzed from the camera 2 or recorder 3 (ST101). At this time, the analysis server 5 can either have already acquired the work log data 12 related to the work process to be analyzed from the management system 4, or acquire it in parallel with the video footage.

[0099] Next, the analysis server 5 performs interaction recognition processing on the video related to the work process (ST102). In the interaction recognition processing, for example, a machine learning model is used to analyze the interaction between the worker and the object (i.e., the worker's movements relative to the object), and the worker's movements and the movement of the object are recognized. As a result, the analysis server 5 can recognize each cycle task included in the work process being analyzed and obtain the time required for each cycle task.

[0100] Subsequently, the analysis server 5 performs work breakdown processing based on the data related to each cycle task and the corresponding work log data (ST103). The work breakdown data generated by the work breakdown processing includes data obtained from the work log 12 and data obtained from video related to the work process.

[0101] Next, the analysis server 5 performs anomaly detection processing based on the work breakdown data (ST104). The anomaly detection processing detects abnormal work in the target work process, and information regarding the detection result is added to the work breakdown data.

[0102] Furthermore, based on the detection results of abnormal work, the analysis server 5 performs a cause analysis of the abnormality and generates a plan for improving the work to prevent the abnormality from occurring (ST105). The data related to the cause analysis and improvement plan obtained in step ST105 is added to the work breakdown data.

[0103] The work breakdown data obtained through this series of processes is ultimately registered in the work process DB26 (ST106).

[0104] Although not shown in the diagram, the analysis server 5 can obtain corrected data by performing the data cleansing process described above on the obtained work breakdown data. The corrected data (work breakdown data after cleansing) is registered in the work process DB 26.

[0105] Next, with reference to Figure 19, the summary generation process by the analysis server 5 will be explained.

[0106] First, the analysis server 5 acquires information for a pre-configured summary period (ST201). Next, the analysis server 5 acquires data (in this case, corrected data) related to the work processes performed during that summary period (ST202).

[0107] Subsequently, the analysis server 5 requests the generation AI server 7 to generate a summary of improvement proposals based on the data it has acquired regarding the work processes performed during the summarization period (ST203). As a result, the analysis server 5 obtains the summary of improvement proposals generated by the generation AI server 7 (ST204).

[0108] The summary data of the improvement proposals obtained through this series of processes is finally registered in the work process DB26 (ST205).

[0109] Next, referring to Figure 20, the data transmission process related to the dashboard display by the analysis server 5 will be explained.

[0110] First, the analysis server 5 receives a request from the user terminal 8 for data display regarding the analysis results (ST301).

[0111] Next, the analysis server 5 determines whether a summary period is set for the request (ST302). If a summary period is set (Yes in ST302), the analysis server 5 executes the summary generation process described above (ST303). On the other hand, if a summary period is not set (No in ST302), the analysis server 5 omits the summary generation process described above.

[0112] Next, the analysis server 5 extracts data on improvement support information for each work process targeted by the request (ST304). In this case, if no summarization period is set (No in ST302), only the data on factor analysis and improvement proposals for each cycle work will be extracted as improvement support information.

[0113] Subsequently, the analysis server 5 transmits the extracted improvement support information data to the user terminal 8 (ST305). As a result, the user terminal 8 displays an improvement support screen based on the improvement support information data.

[0114] Thus, the analysis server 5 (work support device) and its work support method in the information processing system 1 can acquire improvement support information for each cycle work in which abnormal work is detected and for at least one of the work processes including said cycle work, and display this improvement support information on a display device such as a user terminal. This improvement support information is useful for the user to improve their work.

[0115] As described above, embodiments have been explained as examples of the technology disclosed in this application. However, the technology in this disclosure is not limited to these embodiments and can be applied to embodiments that have been modified, replaced, added, or omitted. Furthermore, it is possible to create new embodiments by combining the components described in the above embodiments.

[0116] The work support device and work support method relating to this disclosure have the effect of generating information useful for improving work based on data obtained from video footage of a worker's work, and are useful as a work support device and work support method that support work improvement based on data obtained from video footage of a worker's work.

[0117] 1: Information processing system 2: Camera 3: Recorder 4: Management system 5: Work process analysis server 7: Generation AI server 8: User terminal 11: Communication network 12: Work log 15: Processor 16: Memory 17: Storage 18: Network I / F 21: Log data acquisition 22: Video acquisition 23: Work breakdown 24: Anomaly detection 25: Summary analysis and improvement proposal 26: Work process DB 27: Data cleansing 28: Summary generation 29: Dashboard display 41: Period setting screen 51: Top page of improvement support screen 52: Work selection field 53: Improvement support information field 56: Text 57: Period setting button 58: Details confirmation button 61: Details page of improvement support screen 62: Cycle work display field 63: Video display field 64: Anomaly detection result display field W: Worker

Claims

1. A work support device that supports the improvement of work based on data obtained from video footage of a worker's work, comprising a processor that performs processing to provide the user with work improvement support information, wherein the processor recognizes one or more work units included in a work process based on the video footage, detects abnormal work in each of the work units, and performs dialogue processing based on the detection result of the abnormal work to acquire the improvement support information relating to at least one of the work units in which the abnormal work was detected and the work process including said work units, and displays the improvement support information on a display device.

2. The work support device according to claim 1, wherein in the dialogue processing, the detection result of the abnormal work is input to the generation model, and the improvement support information is obtained based on the output of the generation model.

3. The work support device according to claim 1 or 2, wherein the processor acquires improvement proposals for each work unit as improvement support information.

4. The work support device according to claim 3, wherein the processor acquires improvement proposals for the work process as improvement support information.

5. The work support device according to claim 4, wherein the processor obtains a summary of the proposed improvements for the work of a plurality of work units as proposed improvements for the work of the work process.

6. The work support device according to claim 5, wherein the processor obtains a summary of proposed improvements to the work process corresponding to the work process performed during a predetermined period, and generates a dashboard screen that displays the proposed improvements to the work process.

7. The work support device according to claim 1, wherein the processor generates a dashboard screen that displays information relating to a plurality of work units in which abnormal work was detected, and improvement support information relating to each of the work units.

8. The work support device according to claim 7, wherein the processor displays the corresponding improvement support information on the dashboard screen based on a user's operation on any of the information relating to a plurality of work units.

9. A work support method that performs a process to provide a user with information to support the improvement of the work, based on data obtained from video footage of a worker's work, wherein a computer recognizes one or more work units included in a work process based on the video footage, detects abnormal work in each of the work units, performs a dialogue process based on the detection result of the abnormal work to obtain the improvement support information relating to at least one of the work units in which the abnormal work was detected and the work process including the work units, and displays the improvement support information on a display device.

Images