Display control apparatus, display control method, and program

Abstract

A display control apparatus (10) comprises: an acquisition unit (11) that acquires time-series control data (131) indicating, in association with a time, a device value used for control of a device to be controlled by a PLC (20), and image data (132) indicating a moving image obtained by capturing an object related to the control; and a display control unit (15) that causes a display unit (16) to reproduce the moving image by superimposing a figure having at least one of a position and a size representing a control amount corresponding to the device value on an image that constitutes the moving image and is an image at a time associated with a control value corresponding to the control amount.

Description

Display control device, display control method, and program

[0001] The present disclosure relates to a display control device, a display control method, and a program.

[0002] In the field of FA (Factory Automation), a process of processing an object is carried out by operating equipment. Due to causes such as the occurrence of abnormalities in this process, or for purposes such as improving yield, an operation may be performed to compare and match log data related to the operating equipment and log data related to the object to be processed (see, for example, Patent Document 1).

[0003] Patent Document 1 describes a technique for displaying a device value of a PLC (Programmable Logic Controller) and an image captured by a camera while synchronizing them. Since the device value corresponds to the operation of the equipment controlled by the PLC, according to the technique of Patent Document 1, it is possible to match the operation of the equipment and the object captured.

[0004] Japanese Patent Application Laid-Open No. 2020-13528

[0005] In the technique of Patent Document 1, the device value is displayed as a time-series waveform or as the value of a device included in a ladder diagram. However, the correspondence between the device value displayed in this way and the photographed subject is not obvious even when the device value and the image are simply displayed in parallel. Therefore, the operator who performs the matching operation needs to be familiar with the correspondence in advance or confirm the correspondence separately. Thus, there is room to make the matching operation simpler.

[0006] The present disclosure has been made under the above circumstances, and an object thereof is to make the matching operation using an image of a processing process performed at the FA site simpler.

[0007] To achieve the above objective, the display control device of the present disclosure includes acquisition means for acquiring time-series control data that shows control values ​​used for controlling a controlled device by the control device in relation to time, and moving image data that shows moving images of objects related to control; and display control means for causing a display device to play a moving image by superimposing a figure having at least one of position and size that represents a control quantity corresponding to the control value onto an image that constitutes the moving image and is associated with the control value corresponding to the control quantity at time.

[0008] According to this disclosure, the matching process using images taken of processing steps carried out at the factory automation (FA) site can be made simpler.

[0009]

[0010] The support system according to the embodiment of this disclosure will be described in detail below with reference to the drawings. The support system is a system that assists managers in managing processing steps carried out by controlling controlled equipment in facilities such as factories and plants.

[0011] Embodiment 1. The support system 1000 according to this embodiment supports the administrator's matching work by displaying the progress of control and the progress of result values ​​obtained by image processing of the object of the processing step, each based on a common reference time.

[0012] As shown in Figure 1, the support system 1000 includes a display control device 10 that controls the content of information displayed to the administrator, a PLC (Programmable Logic Controller) 20 that controls the controlled devices, a servo amplifier 21 and a servo motor 22 which are the controlled devices, and cameras 31 and 32 that photograph the workpiece 40 which is the target of processing.

[0013] The PLC 20 is also called a programmable controller. The PLC 20 controls the servo amplifier 21 and the servo motor 22 using device values ​​211 stored in its built-in memory 210. The device values ​​211 are values ​​input to the PLC 20 from an external device including the servo amplifier 21, intermediate values ​​used in calculation processing by the PLC 20, or values ​​output from the PLC 20 to an external device including the servo amplifier 21. For example, if the PLC 20 outputs the result of calculations based on the sensing result of a sensor (not shown) as a command value to the servo amplifier 21, the sensing result, the intermediate value used in the calculation, and the command value can each be the device value 211. Since the device values ​​211 represent the control state in this way, the log of device values ​​211 is used to represent the control history. The device values ​​211 are an example of control values ​​used to control controlled equipment.

[0014] The servo amplifier 21 and servo motor 22 process the workpiece 40 under the control of the PLC 20. The processing of the workpiece 40 may be transport, machining, assembly, or other processes. The servo amplifier 21 is connected to the PLC 20 via a signal line and to the servo motor 22 via a signal line and a power line.

[0015] The following description will focus on an example where the processing of workpiece 40 involves placing workpiece 40 on top of a different workpiece 40a, as shown in Figure 2. Workpiece 40 is transported by an arm 23 attached to a servo motor 22, as indicated by the thick arrow in Figure 2, and is positioned so that the mark 41 on workpiece 40 overlaps with the mark 41a on workpiece 40a in the vertical Z-axis direction. This process is performed sequentially for a large number of workpieces 40, but abnormalities may occur where the process does not complete as intended due to factors such as printing defects in marks 41, 41a, processing defects in workpieces 40, 40a, or interference between the arm 23 and workpieces 40, 40a. When such abnormalities occur, the cause of the abnormality is investigated and resolved by comparing the log of device values ​​211 with the captured image of workpiece 40. Some or all of the workpieces 40, 40a and marks 41, 41a correspond to examples of objects to be photographed.

[0016] Returning to Figure 1, the workpiece 40 to be processed is repeatedly photographed from different angles by cameras 31 and 32, as shown in Figure 1. The images obtained from the photography are transmitted to the display control device 10 via the communication line 301. The communication line 301 is, for example, a USB (Universal Serial Bus) cable or a Coaxpress-compatible cable. Alternatively, the images may be transmitted via a network such as a LAN (Local Area Network) instead of the communication line 301.

[0017] Cameras 31 and 32 simultaneously capture images when they receive I / O (Input / Output) signals transmitted from the PLC 20 via the communication line 202. Since cameras 31 and 32 capture images triggered by a common I / O signal, the timing of each camera's capture is synchronized. Furthermore, if the I / O signal is transmitted by the PLC 20 when reading device values ​​and creating a log, the timing of image capture and the timing of device value recording will be synchronized.

[0018] The display control device 10 is a UI (User Interface) terminal, such as an industrial PC (Personal Computer), and is connected to the PLC 20 via a communication line 201, such as a USB cable. The display control device 10 functions as an engineering tool for creating and editing control programs that define the control content of the PLC 20, and may write the control program to the PLC 20 and have the PLC 20 execute it.

[0019] The display control device 10 has the hardware configuration shown in Figure 3. Specifically, the display control device 10 is configured as a computer having a processor 101, a main memory unit 102, an auxiliary memory unit 103, an input unit 104, an output unit 105, and a communication unit 106. The main memory unit 102, the auxiliary memory unit 103, the input unit 104, the output unit 105, and the communication unit 106 are all connected to the processor 101 via an internal bus 107.

[0020] The processor 101 includes a CPU (Central Processing Unit) as a processing circuit. The processor 101 performs various functions and processes described later by executing a program P1 stored in the auxiliary storage unit 103. Program P1 is an example of a display control program.

[0021] The main memory unit 102 includes RAM (Random Access Memory). Program P1 is loaded into the main memory unit 102 from the auxiliary memory unit 103. The main memory unit 102 is then used as a working area for the processor 101.

[0022] The auxiliary storage unit 103 includes non-volatile memory such as EEPROM (Electrically Erasable Programmable Read-Only Memory) and HDD (Hard Disk Drive). In addition to program P1, the auxiliary storage unit 103 stores various data used in the processing of the processor 101. The auxiliary storage unit 103 supplies data to be used by the processor 101 according to the instructions of the processor 101. The auxiliary storage unit 103 also stores data supplied from the processor 101.

[0023] The input unit 104 includes input devices such as hardware switches, input keys, keyboards, and pointing devices. The input unit 104 acquires information entered by the user of the display control device 10 and notifies the processor 101 of the acquired information.

[0024] The output unit 105 includes output devices such as an LED (Light Emitting Diode), an LCD (Liquid Crystal Display), and a speaker. The output unit 105 presents various information to the user according to instructions from the processor 101.

[0025] The communication unit 106 includes a communication interface circuit for communicating with an external device. The communication unit 106 receives a signal from an external source and outputs data indicated by this signal to the processor 101. The communication unit 106 also transmits a signal indicating the data output from the processor 101 to the external device. Although one communication unit 106 is typically shown in Figure 3, the display control device 10 may have multiple communication units 106. For example, the display control device 10 may have a separate communication unit 106 for communication via the communication line 201 in Figure 1 and a separate communication unit 106 for communication via the communication line 301 in Figure 1.

[0026] The aforementioned hardware configuration works in cooperation to enable the display control device 10 to perform various functions. Specifically, as shown in Figure 4, the display control device 10 includes an acquisition unit 11 that acquires control data 131 from the PLC 20 and stores it in the storage unit 13, and an acquisition unit 132 that acquires image data 132 from the cameras 31 and 32 and stores it in the storage unit 13; a reception unit 12 that receives parameters used for image processing from the user; a storage unit 13 that stores various information; an image processing unit 14 that performs image processing on the image data 132; a display control unit 15 that controls the display content of the display unit 16; and a display unit 16 that displays information to the user. Note that the arrows connecting each functional unit in Figure 4 only indicate the main information transmission paths, and information can also be transmitted through paths not shown in Figure 4, as will be described later.

[0027] The acquisition unit 11 is primarily realized through the cooperation of the processor 101 and the communication unit 106. The control data 131 acquired by the acquisition unit 11 is time-series data of device values ​​211 with timestamps, as illustrated in Figure 5. In the example in Figure 5, the timestamp, one or more device values ​​to be collected, and an imaging trigger indicating whether or not an I / O signal was transmitted when the device value was read from the memory 210 are associated with each other. In the figure, an imaging trigger value of "1" indicates that there was an I / O signal as an imaging trigger, and an imaging trigger value of "0" indicates that there was no I / O signal.

[0028] Furthermore, the image data 132 acquired by the acquisition unit 11 is time-series data that shows the relationship between the captured image and the frame count, as illustrated in Figure 5. In the example in Figure 5, an identifier for identifying the camera that took the image is shown in parentheses. For example, camera

[31] represents camera 31 whose identifier is "31". The identifier and the reference numeral in the figure are the same.

[0029] As described above, since an image is captured at the same time that the device value associated with the imaging trigger, which is "1", is recorded, the synchronized device value and image are identified by this imaging trigger. Specifically, the device value with the first imaging trigger "1" in the control data 131 and the first image that constitutes the image data 132 are identified as corresponding in that order. The acquisition unit 11 is an example of an acquisition means that acquires time-series control data that shows control values ​​used for controlling the controlled device in relation to time, and image data that includes images of objects related to control that are repeatedly captured at timings corresponding to the time associated with the control value.

[0030] Returning to Figure 4, the reception unit 12 is mainly implemented by the input unit 104. The reception unit 12 receives image processing parameters 133 used for image processing, conversion parameters 134 for converting pixel units to physical quantity units, and range parameters 135 indicating the range of the position of the workpiece 40 as a physical quantity from the user, and stores these received parameters in the storage unit 13. The reception unit 12 corresponds to an example of a first reception means that receives conversion parameters for converting the units of pixels constituting an image to physical quantity units. Note that, as will be described later, if the conversion to physical quantity units is not performed, the reception unit 12 may omit receiving the conversion parameters 134 and range parameters 135.

[0031] The memory unit 13 is primarily implemented by at least one of the main memory unit 102 and the auxiliary memory unit 103.

[0032] The image processing unit 14 is primarily implemented by the processor 101. The image processing unit 14 performs image processing, such as pattern matching, on the image contained in the image data 132. The image processing unit 14 obtains result values ​​regarding the position of the workpiece 40 in the image by performing image processing based on the image processing parameters 133. These result values ​​are, for example, at least one of the following values: position, velocity, acceleration, orientation, angular velocity, angular acceleration, and area of ​​the workpiece 40 itself or a part of the workpiece 40 estimated from the image, or a value indicating the shape. The value indicating the shape is, for example, the degree of agreement with a pre-registered 3D model shape or 2D silhouette shape.

[0033] The image processing unit 14 may obtain the result value after estimating the position of the workpiece 40, or it may obtain the result value without estimating the position of the workpiece 40. For example, the image processing unit 14 may directly estimate the velocity or acceleration of the workpiece 40 as a result value without specifying the position of the workpiece 40. Since these velocities and accelerations are derived from the first and second derivatives of the position of the workpiece 40, they can be considered result values ​​relating to the position of the workpiece 40. Furthermore, since the posture and shape of the workpiece 40 depend on the position of each part of the workpiece 40, they can be considered result values ​​relating to the positions of these parts. In addition, if the workpiece 40 has movable parts or if the object of the processing process is a powder or a fluid, the area of ​​the object is determined by the position of each part of the object. For this reason, the area can be considered a result value relating to the position of the parts of the object.

[0034] Furthermore, the image processing unit 14 may obtain result values ​​in pixel units, or it may obtain result values ​​in physical units by converting from pixel units to physical unit units using the conversion parameter 134. That is, the result values ​​may be expressed in pixel units or in physical units. Physical quantities refer to the attributes, states, or properties that an object actually possesses, which can be measured by methods other than optical methods, regardless of whether a camera is present or not. The units of physical quantities are, for example, units of the International System of Units using meters and seconds, or units following the imperial system.

[0035] The following section will primarily describe an example of performing image processing using the conversion parameter 134 to obtain result values ​​for physical quantities. Details of the image processing will be described later. The image processing unit 14 corresponds to an example of an image processing means that obtains result values ​​related to the position of an object by processing the image.

[0036] Furthermore, the image processing unit 14 corrects the trend of the result values ​​based on the range parameter 135. For example, if the movement speed of the workpiece 40 estimated by the image processing unit 14 through image processing exceeds the upper limit indicated by the range parameter 135, the image processing unit 14 corrects the trend of the result values ​​indicating position or speed so that they do not exceed the upper limit.

[0037] The display control unit 15 is primarily implemented by the processor 101. The display control unit 15 acquires the changes in result values ​​from the image processing unit 14 and reads control data 131 from the storage unit 13. The display control unit 15 then displays the changes in result values ​​and device values ​​on the display unit 16 in a manner that correlates these changes. More specifically, the display control unit 15 superimposes graphs showing these changes onto a region defined by a common time axis and draws them on the display unit 16. Details of the drawing by the display control unit 15 will be described later. The display control unit 15 is an example of a display control means that displays the changes in control values ​​indicated by control data and the changes in result values ​​on a display device in a manner that correlates them based on a common reference time.

[0038] The display unit 16 is primarily implemented by the output unit 105. The display unit 16 displays information to the user according to the instructions of the display control unit 15. The display unit 16 is an example of a display device.

[0039] Next, the display control processing performed by the display control device 10 will be described in detail using Figures 6 to 19. The display control processing in Figure 6 is started when the user performs a specific operation on the application software of the display control device 10. This application software may be the engineering tool described above.

[0040] As shown in Figure 6, in the display control process, the acquisition unit 11 acquires control data 131 and image data 132 (step S1) and stores this data in the storage unit 13. Next, the reception unit 12 receives image processing settings from the user and acquires image processing parameters 133 (step S2).

[0041] Figure 7 shows an example screen for configuring image processing settings. In the screen shown in Figure 7, when a camera is selected from the list 501 and the "Capture Image" button 502 is pressed, the reception unit 12 works in cooperation with the acquisition unit 11 to cause the selected camera to capture a new image, and the captured image is displayed in the image display area 520 in real time. Also, when the "Load Image File" button 503 is pressed, the reception unit 12 reads the image specified by the user from the storage unit 13 and displays the read image in the image display area 520.

[0042] Furthermore, the reception unit 12 accepts the selection of one of the filters from list 51 as a specification for image processing preprocessing. This list 51 includes "Gaussian", "Median", "Average", "Maximum", and "Minimum".

[0043] The reception unit 12 also accepts the selection of an image processing mode from the list 52. This image processing mode can be, for example, edge detection, circle detection, line detection, angle detection, color detection, normalized correlation, pattern matching, blob detection, feature matching (matching method, optical flow method), or one of the calculation modes described later. In accordance with the selected image processing mode, the reception unit 12 displays an input field 54 for the parameters handled by that image processing mode.

[0044] In addition, the reception unit 12 displays the result of attempting image processing using the temporarily designated image and parameters in the area 511. Specifically, when updating the display content in the image display area 520 and when changing the value of the parameter input in the input field 54, the reception unit 12 causes the image processing unit 14 to attempt image processing and displays the result of the attempt in the area 511. As a result, in the area 511, the position of the workpiece 40 or a part of the workpiece 40, or the result of intermediate processing represented by filter processing, edge point recognition, and straight line conversion processing, is enlarged and displayed in the form of a figure or polygon. When the image processing mode is pattern matching, the reception unit 12 displays the image of the model registered from the input field 54 in the area 512 adjacent to the area 511.

[0045] In addition, when the button 53 for "parameter automatic adjustment" is pressed, the reception unit 12 adjusts the parameters by causing the image processing unit 14 to attempt image processing on the designated image while changing the value of the parameter to search for an appropriate value. The parameters to be adjusted are parameters for which a numerical value should be specified, other than parameters indicating a range. In an image processing mode in which a matching rate can be calculated, such as pattern matching and the detection of a figure such as a rectangle or a circle or a color, the parameter is adjusted to the value at which the matching rate is maximized. Also, when the result of the image processing trial is an estimated value of the position of the workpiece 40 or a part thereof, the parameter is adjusted to a value that meets these conditions prioritized in the order of "detected", "few over-detections", "small change amount of the estimated value when changing each parameter value to a nearby value of large or small", and "short processing time". The nearby value of the parameter value is a value for which the difference from the parameter value, or the ratio of the difference to the parameter value, is a predetermined specified value.

[0046] In addition, the reception unit 12 displays the result of the image processing trial in the result column 55. Specifically, the reception unit 12 displays a numerical value or a character string indicating the result of the image processing and the time duration taken for the image processing, and when the matching rate is calculated, the calculated matching rate is also displayed.

[0047] FIG. 8 shows an example of a screen when the button 504 for "Live Image Processing" is pressed. At this time, shooting by the selected camera is repeatedly executed in real time, and trials of image processing for each captured image are continuously executed. The trial results of the image processing are displayed in the graph display area 530. In the example of FIG. 8, the transitions of the X coordinate and the Y coordinate in the image of the pattern detected by pattern matching are displayed as graphs, respectively. Further, when the check box 521 for "Trajectory Display" is checked, the reception unit 12 draws in the image display area 520 so that the trajectory of the detected pattern is shown by a thick line.

[0048] FIG. 9 shows an example of a screen when the button 506 for "Collected Data Image Processing" is pressed and any one of a plurality of image data 132 is selected from the list 505. At this time, batch image processing is tried on the image data 132 showing time-series images, and the result is displayed in the graph display area 530. When the check box 521 is checked, the trajectory of the detection target detected in the image display area 520 is drawn in the same manner as in the example of FIG. 8.

[0049] In FIG. 5, the single image data 132 was shown as including time-series images captured by a plurality of cameras 31 and 32, but the image data 132 may be time-series images captured by a single camera. Further, time-series images captured by the same camera at different periods may be treated as different image data 132, respectively. The selection of the image data 132 using the list 505 in FIG. 9 corresponds to the selection of the image data 132 captured by a single camera during a specific period.

[0050] Furthermore, when the knob on the bar 522 is operated, the reception unit 12 displays an image associated with the time corresponding to the position of the knob in the image display area 520, and moves the line 531 in the graph display area 530 to correspond to the position of the knob. Similarly, when the position of the line 531 is operated along the horizontal axis of the graph, i.e., along the time axis, the reception unit 12 updates the image in the image display area 520 to correspond to the position of the line 531, and changes the position of the knob on the bar 522.

[0051] Furthermore, in the input field 513 for "search area tracking," if the check box indicating whether or not to execute is checked and processed data showing the image processing result is selected as the tracking target from the list, the reception unit 12 sets a search area for searching for the detection target based on the partial region in the image of the selected processed data where the detection target was detected. Search area tracking is used, for example, in pattern matching when it is expected that the pattern to be detected from the image data 132 will appear in roughly the same position as other image data 132, in order to reduce the computational load.

[0052] Furthermore, the target of the search area tracking is not limited to processed data that shows the results of image processing that has already been performed, but may also be scheduled data for which image processing is planned. If scheduled data is selected as the target of tracking, image processing related to the scheduled data will be performed preferentially, and then a search area based on the results of that image processing will be set, and image processing will be performed on the image data 132 selected from list 505. In addition, in order to avoid circular definitions of the target of tracking, it is not possible to specify image data 132 selected from list 505 as the target of tracking for the search area for image processing related to scheduled data.

[0053] For the image data 132 selected on the screen in Figure 9, data showing the result of image processing using the input parameters may become new processed data or scheduled data. The names of this processed data and scheduled data are specified in the input field 50. Furthermore, when image data 132 is selected from the list 505 and the "Save" button 56 is pressed, processed data or scheduled data based on the image data 132 is registered, and this registered data may be the subject of the graph display described later.

[0054] In addition to processed data and scheduled data, users can also select "previous image processing result" or "previous image difference" as the target for tracking. If "previous image processing result" is selected, the search area for the current image is set to track the image processing result of the most recent image from the series of images included in the image data 132 selected from list 505. If "previous image difference" is selected, the search area is set to the region where a difference exceeding a threshold occurs when comparing the previous image and the current image. If there are multiple regions where a difference occurs, multiple search areas including each region are set and image processing is performed.

[0055] If, during tracking of the search area, the target to be detected is not found, or if detection fails due to factors such as deformation of the workpiece 40, the search area may be reset to the entire image and the image processing may be performed again.

[0056] Figure 10 shows an example screen when the "Calculation" mode is selected as the image processing mode from List 52. In this case, the result of the calculation shown in the calculation formula entered in input field 58 is applied to the image processing result values ​​included in the processed data displayed in List 57, and this result is the processing result of the "Calculation" mode. In the figure, a, c, and e represent the X coordinate values ​​obtained as a result of the image processing of each processed data, and b, d, and f represent the Y coordinate values. By appropriately defining the calculation formula, the distance between patterns, the center point, the centroid, or the inclination can be easily obtained. Calculation formulas can be added or deleted using the "Add" and "Delete" buttons in input field 58.

[0057] When a calculation formula is entered in the input field 58 and the "Execute Image Processing" button is pressed, the result calculated by the entered formula is displayed directly below the input field. Also, when the "Live Image Processing" button 504 or the "Collected Data Image Processing" button 506 is pressed, the progress of the calculation result from the entered formula is displayed in the graph display area 530.

[0058] As explained using Figures 7 to 10, the system accepts the setting of image processing parameters 133 while attempting image processing as needed, and registers the combination of the accepted image processing parameters 133 and the image data 132 on which image processing should be performed using those parameters 133. Hereafter, this combination of image processing parameters 133 and image data 132 may be referred to as image processing data. The image processing data may be the processed data or scheduled data described above.

[0059] Returning to Figure 6, following step S2, the reception unit 12 receives the graph display settings and obtains the conversion parameter 134 and the range parameter 135 (step S3).

[0060] Figure 11 shows an example of a screen displayed when accepting graph display settings. On this screen, settings are entered for displaying two graphs superimposed in one or more areas. Specifically, settings for "Graph No. 1" and "Graph No. 2" to be superimposed in "Graph Area A1" and settings for "Graph No. 1" and "Graph No. 2" to be superimposed in "Graph Area A2" are entered. For each graph, the image processing result or control data 131 is selected from list 611. The image processing results that can be selected are the processed data or planned data registered in Figures 7 to 10. For example, "Image Processing Setting B1 (X)" in Figure 11 shows the change in the X coordinate value in the image, which is output when image processing is performed with the parameter value registered under the name "Image Processing Setting B1". Also, "Axis Ax.1 - Current Value" and "Axis Ax.2 - Current Value" in Figure 11 show the change in the device value registered under these names. When comparing control data 131 and image data 132, typically, one of the image processing data is selected from list 611 as one of the graphs to be displayed superimposed, and the other graph is selected from list 611 as the trend of a device value.

[0061] Furthermore, the type of result value to be displayed is selected from List 612. For example, if any image processing result is selected from List 611 and then "velocity" is selected from List 612, the velocity obtained by differentiating the position as a result of image processing will be displayed on the graph. The unit of the value to be displayed on the graph is selected from List 613. For example, if "position" is selected from List 612 and then "mm" is selected from List 613, the position indicated by the number of pixels as a result of image processing will be converted to the position of the workpiece 40 in millimeters and displayed as a graph.

[0062] Furthermore, if the "Noise Reduction" check box 601 is checked for each graph, the noise reduction process described later will be executed. Also, if the "Inter-Graph Fitting" check box 602 is checked for each graph area, the fitting process described later will be executed for the two graphs displayed in that area.

[0063] When the "Data Settings" button 64 is pressed on the screen shown in Figure 11, the reception unit 12 transitions to the settings screen shown in Figure 12. At the top of the settings screen in Figure 12, the image processing data and control data 131, which indicate the settings for image processing, are set by checking or unchecking the check boxes to determine whether they should be selected as candidates in the list 611 of Figure 11. At the bottom of the settings screen in Figure 12, the resolution of each camera 31, 32 is set. This resolution represents the ratio of the number of pixels between two pixels that make up the captured image to the distance between the parts of the captured workpiece 40 corresponding to each pixel. This resolution is used when a physical unit different from the number of pixels is selected from the list 613 of Figure 11. Specifically, if "None" is selected from the list 613 of Figure 11, a graph in pixel count units is displayed, and if "cm", "mm", or "um" is selected, a graph of physical units converted from the number of pixels according to the set resolution is displayed. The following describes an example where "cm", "mm", or "um" is selected and the conversion from the number of pixels to physical units is performed. Resolution is an example of a conversion parameter.

[0064] Furthermore, the "motor range parameter" shown in Figure 12 is an example of a range parameter, and the value of this range parameter is received by the reception unit 12.

[0065] Returning to Figure 11, when the "Save Settings Data" button 62 is pressed, the settings configured in steps S1 to S3 are saved along with the identification name. Also, when the "Load Settings Data" button 63 is pressed, the user can specify an identification name, and the settings saved under that identification name will be loaded. Once the image processing settings and graph display settings are complete, the user presses the "Start Data Display" button 61.

[0066] Returning to Figure 6, following step S3, the image processing unit 14 performs image processing according to the settings (step S4), and if necessary, converts the values ​​obtained as a result of the image processing into physical quantities. For example, as illustrated in Figure 13, the image processing unit 14 sequentially detects the detection target by pattern matching and converts the pixel-level position of the detected pattern into a physical unit position according to the set resolution. In the example in Figure 13, for the X and Y coordinate values ​​corresponding to each timestamp, the pixel-level position indicated by "px" is converted into a micrometer-level position indicated by "um". Note that if the results of the image processing trial performed during the information setting in steps S1 and S2 can be reused, the image processing in step S3 may be omitted.

[0067] Next, the image processing unit 14 performs differential processing and noise reduction processing according to the settings (step S5). In Figure 11, if a type that does not require differential processing is selected from list 612 for all graphs, the differential processing is omitted. Also, in Figure 11, if it is specified in check box 601 that noise reduction should not be performed for all graphs, the noise reduction processing is omitted.

[0068] Differential calculus is the process of determining velocity or acceleration from a time series of position data. The central difference method is used to derive velocity. Specifically, the derivative obtained by differentiating the value immediately before and immediately after a given time in the time series data is taken as the velocity at that particular time. For example, if the position at time t is x(t), the velocity at time t is calculated from x(t-1) and x(t+1).

[0069] For the conversion to acceleration, the above-described differentiation operation is performed twice on the value at a specific time in the time series data, as well as the values ​​immediately before and after it. For example, the acceleration at time t is calculated by differentiating the velocity obtained by differentiating x(t) and x(t-1), and then further differentiating the velocity obtained by differentiating x(t) and x(t+1). In the differentiation process, in order to avoid the occurrence of time-direction discrepancies before and after the process, the derivative value at a specific time is calculated from the values ​​immediately before and after that specific time, as described above.

[0070] In the noise reduction process, as shown in Figure 14, noise components are removed by low-pass filtering. While methods such as moving average, frequency space cutoff, and Gaussian convolution can be used for noise reduction, the moving average method is employed, which calculates the average of N values ​​before and after a specific time point to obtain the noise-reduced value at that point. For example, if N is 3, the moving average of x(t-1), x(t), and x(t+1) becomes the noise-reduced value at time t. The value of N may be set by the user or predetermined. Similar to the differentiation process described above, in the noise reduction process, to avoid time-direction shifts before and after processing, the processed value at a specific time point is calculated from values ​​before and after that specific time point.

[0071] Returning to Figure 6, following step S5, the display control unit 15 performs inter-graph fitting (step S6). Note that if inter-graph fitting is not specified in Figure 11, step S6 is omitted. In inter-graph fitting, in detail, as shown in Figure 15, it is set that the graphs should be displayed in the same area, and the display control unit 15 sets two analysis points for the transitions of each of the two values ​​obtained in steps S1 to S5. The analysis points are placed at predetermined fixed intervals, but the time corresponding to the analysis points may be specified by the user.

[0072] Next, each graph representing the trend of values ​​is converted into a linear graph that takes the value at the analysis point. Then, using the linear graph with the smallest slope as a reference, the slope of the reference linear graph is changed and the scale is adjusted so that the distances d1 and d2 between the graphs at the analysis point match. Furthermore, the offset value of the reference graph is adjusted so that the scaled graphs overlap as much as possible. When overlapping graphs, the least squares method can be used. Figure 16 shows an example of evaluating the distance between two graphs using the least squares method. The reference graph is moved vertically, and the sum of the squares of the values ​​at each time point is taken as the distance between the graphs, and the offset value that minimizes this distance is derived. By adjusting the scale and offset value of the graphs in this way, the user's task of comparing two graphs becomes easier.

[0073] Returning to Figure 6, following step S6, the display control unit 15 displays the image and graph on the display unit 16 (step S7). Specifically, it displays a screen on the display unit 16 as illustrated in Figure 17. On this screen, the image contained in the image data 132 is displayed in the image display area 71, and the detected object detected from this image is also displayed with a dashed outline frame indicating the detection result superimposed on the image.

[0074] Furthermore, the graph display area 72 displays graphs showing the changes in result values ​​obtained through image processing, conversion to physical quantities, differentiation processing, and noise reduction processing, and graphs showing the changes in device values, superimposed on a common time axis. The graphs displayed in the graph display area 72 can be enlarged or reduced horizontally or vertically using mouse and keyboard operations. When the "Add Graph" button 701 is pressed, a new graph area is added to the graph display area 72, and when the "Delete" button 702 is pressed, the corresponding graph area is deleted. This allows for further editing of the displayed graphs while they are still being displayed. In addition, whether or not a graph is displayed is determined by whether or not the check box 721 adjacent to the name of each graph is checked.

[0075] The reception unit 12 corresponds to an example of a fourth reception means that receives the specification of control data and image data from which a graph should be created from multiple control data and multiple image data, as well as the specification of whether or not to display the graph. The image processing unit 14 corresponds to an example of an image processing means that obtains result values ​​from the specified image data received by the fourth reception means, and the display control unit 15 corresponds to an example of a display control means that switches whether or not to display the graph according to the specification received by the fourth reception means.

[0076] Furthermore, similar to the example shown in Figure 9, the knobs on bar 711 and the lines 703 in the graph area are linked, and images of the time corresponding to these knobs or lines 703 are displayed in the image display area 71. The images in the image display area 71 can be played, reversed, and paused as moving images by operating button 712. Also, the images in the image display area 71 can be advanced or rewinded by operating button 713.

[0077] A flag 714 is set on bar 711. Flag 714 is assigned to the time when the difference between two trends displayed in the same graph area exceeds a predetermined threshold. When the cursor is hovered over flag 714 and selected, the image for that time is displayed in the image display area 71, and the line 703 in the graph display area 72 also moves to the position corresponding to that time. Flag 714 is set by pressing the "Set Flag" button 715.

[0078] When button 715 is pressed, the screen shown in Figure 18 is displayed. This setting screen allows information to be set in two setting modes: "Command Value Interval" and "Value Interval". The "Command Value Interval" mode is a mode in which a threshold is set in order to assign a flag 714 to the time of the command value when the time difference between the rising or falling edges of the command value (device value) and the result value obtained by image processing, i.e., the distance along the horizontal axis of the graph, is greater than the threshold. If the transition of the command value (device value) is not handled, no information is set in the "Command Value Interval" mode. On the other hand, the "Value Interval" mode is a mode in which a threshold is set in order to assign a flag 714 to the time when the difference between the values ​​between two transitions, i.e., the distance along the vertical axis between the graphs, is greater than the threshold.

[0079] In Figure 18, the threshold value, expressed in "%", indicates the percentage relative to the overall average difference at which the object should be detected. The "search range" is the search range for checking the difference, and in "command value interval" mode, it represents the distance in the time direction, while in "value interval" mode, it represents the distance in the direction of magnitude of the values. If no object to be detected is found within this search range, the flag 714 is not assigned, and the object is not included in the calculation of the average difference.

[0080] Note that Figure 17 shows an example where there is one image data 132 and one image is reproduced in the image display area 71, but it is not limited to this. If multiple image data 132 are set to be the target of image processing, multiple images may be displayed in the image display area 71 as shown in Figure 19.

[0081] As described above, the display control device 10 obtains the changes in the resulting value related to the position of the workpiece 40 from an image of the workpiece 40, and displays the changes in the resulting value and the changes in the device value in a manner that correlates them based on a common reference time. Here, correlation based on a common reference time means that the reference point and time scale in the time direction coincide, and that at any point in the overlapping period of the two changes, the time at which the values ​​were generated coincides with each other. This makes it easy to match the state of an object with the control history. Therefore, the efficiency of the matching work using images of processing steps performed on the factory automation site can be improved.

[0082] The conversion parameter 134 indicates the relationship between the number of pixels from one pixel to another that constitute the image, and the distance from the part of the workpiece 40 corresponding to one pixel to the other part corresponding to the other pixel when the workpiece 40 is captured in the image. Furthermore, the resulting value regarding the position of the workpiece 40 includes at least one value from among the position, velocity, acceleration, orientation, angular velocity, angular acceleration, and area of ​​the workpiece 40, or a value indicating its shape. This allows the user to directly determine whether the processing status of the workpiece 40 progressed as intended.

[0083] Furthermore, the control data 131 may also be converted into a value relating to position as a physical quantity. For example, if the control data 131 is set to be in units of travel distance from the list 613 in Figure 11, the travel distance of the workpiece 40 when the servo motor 22 rotates once, and the "pulse / rev" value indicating the number of pulses per rotation, which are set as shown at the bottom of Figure 12, may be used to obtain the change in the resulting value relating to the position of the arm 23 from the device value representing the number of pulses, and this change may be displayed.

[0084] Furthermore, the display control unit 15 superimposed the two trends as graphs in a graph area defined by a common time axis. This allows the user to visually grasp the trends and easily perform the comparison work. The graph showing the trend of device values ​​corresponds to an example of the first graph, and the graph showing the trend of result values ​​obtained by image processing corresponds to an example of the second graph.

[0085] Furthermore, the display control unit 15 compares the changes in device values ​​with the changes in result values ​​obtained through image processing, and adjusts at least one of the scale and offset values ​​for one of the two graphs before displaying it. This makes the graphs appear to have similar shapes, facilitating visual comparison between them. Note that the scale and offset values ​​may be adjusted for both graphs.

[0086] Furthermore, the reception unit 12 receives a range parameter 135 indicating the range of the physical position of the workpiece 40, the image processing unit 14 corrects the image processing result based on the range parameter 135, and the display control unit 15 displays a graph obtained by correcting the transition of the result value based on the range parameter 135. As a result, even if an unrealistic movement of the workpiece 40 is obtained as a result of image processing due to factors such as noise contained in the image, a graph is displayed that brings that movement within a realistic range. The reception unit 12 corresponds to an example of a third reception means for receiving range parameters.

[0087] Although Embodiment 1 has been described above, Embodiment 1 may be modified as follows.

[0088] For example, as shown in Figure 20, if the number of patterns detected from the image becomes multiple and the number of graphs increases, it can become a complicated task to determine which pattern each graph corresponds to. Therefore, the display control unit 15 may indicate the correspondence between patterns and graphs using lines 81 as shown in Figure 20.

[0089] In the example shown in Figure 20, the reception unit 12 corresponds to an example of a second reception means that receives a specified time using a knob on line 703 or bar 711. The display control unit 15 corresponds to an example of a display control means that displays an image associated with the specified time along with the first graph and the second graph, and when a result value is obtained from a sub-region corresponding to a pattern contained in the displayed image, it highlights and displays the sub-region in association with the second graph.

[0090] Furthermore, while an example has been described in which values ​​in pixel units are converted into result values ​​that represent the position of the workpiece 40 in physical units based on conversion parameters, the method is not limited to this. For example, even if the conversion parameter 134 is not given, if the 3D model shape of the workpiece 40 is given in advance, physical quantities of the workpiece 40, such as velocity or acceleration, can be obtained from the image by fitting the projected image of the workpiece 40 in the image with the 3D model shape. Also, even without prior information, if one or both of the cameras 31 and 32 are depth cameras, physical quantities can be obtained directly from the image.

[0091] Embodiment 2. Next, Embodiment 2 will be described, focusing on the differences from Embodiment 1 described above. Note that the same or equivalent components as in Embodiment 1 will be referred to by the same reference numerals. This embodiment is similar to Embodiment 1 in that it displays information for comparing and verifying control data 131 and image data 132, but the content of the displayed information is different. In detail, this embodiment differs from the above embodiment in that it displays a figure obtained by processing the control value contained in the control data 131 superimposed on the image of the image data 132.

[0092] As shown in Figure 21, the PLC 20 according to this embodiment is a control device that controls the machine end by controlling the controlled device 220. The controlled device 220 is a motor that rotates the support base 221 on which the workpiece 42 is placed, and actuators 222 and 223 that move the support base 221 in the X-axis and Y-axis directions, respectively. The PLC 20 repeatedly records device values ​​corresponding to at least one of the command values ​​for the motor and actuators 222 and 223, intermediate values ​​used in calculations to obtain the command values, and other values ​​used in control processing. These device values ​​correspond to at least one control quantity of the support base 221 as the machine end, i.e., the workpiece 42, such as position, velocity, acceleration, attitude, angular velocity, and angular acceleration. For example, the number of pulses indicated by the device value is proportional to the position of the actuator 222, or corresponds one-to-one with that position. The PLC 20 then transmits control data showing the time-series device values ​​to the display control device 10.

[0093] Camera 30 captures still images continuously, including the workpiece 42 and the support base 221 within its shooting range. Image data, including the captured time-series images, is transmitted from camera 30 to the display control device 10.

[0094] The display control device 10 calculates a control quantity corresponding to a device value indicated by control data, and displays a figure having a position or size corresponding to the calculated control quantity by superimposing it onto an image included in the image data. In the example in Figure 21, an arrow indicating the movement of the workpiece 42 on the XY plane is displayed to the user superimposed on the image.

[0095] The display control device 10 has the functional configuration shown in Figure 22. In detail, the display control device 10 includes an acquisition unit 11 that acquires control data 131 and image data 132, a reception unit 12 that receives conversion parameters 136 for converting device values ​​into control quantities and display parameters 137 related to the information to be displayed, a storage unit 13, an image processing unit 14, a control data processing unit 17 that obtains a time-series control quantity by processing the control data 131 based on the conversion parameters 136, a display control unit 15 that overlays a figure corresponding to the control quantity onto the image according to the display parameters 137 and displays it on the display unit 16, and a display unit 16.

[0096] The acquisition unit 11 acquires control data 131 and image data 132, which include device values ​​and images, respectively, that have been collected synchronously. The image data 132 may be in video format. Image data 132 is an example of video data. The acquisition unit 11 is an example of an acquisition means that acquires time-series control data that shows control values ​​used for controlling controlled equipment by a control device in relation to time, and video data that shows video images of objects related to control.

[0097] The reception unit 12 receives the conversion parameters 136 and display parameters 137 input via the user interface described later, and stores these received parameters in the storage unit 13.

[0098] The image processing unit 14 may perform image processing equivalent to that in Embodiment 1, or it may perform other image processing. For example, the image processing unit 14 may detect the workpiece 42 in the image by pattern matching and track the movement of the workpiece 42 within the image.

[0099] The control data processing unit 17 is primarily implemented by the processor 101. In order to display a figure of an appropriate size on the display screen of the display unit 16, the control data processing unit 17 converts the unit of the control quantity to the unit of pixels for each time-series control quantity indicated by the control data 131. Details of this conversion will be described later.

[0100] The display control unit 15 displays a figure having a position or size corresponding to a control quantity whose units have been converted, superimposed on an image of the same time, and plays back the moving image shown by the image data 132 according to the user's operation. The display control unit 15 is an example of a display control means that superimposes a figure having at least one of a position and size representing a control quantity corresponding to a control value onto an image that constitutes the moving image, which is an image of the time associated with the control value corresponding to the control quantity, and plays back the moving image on the display device.

[0101] Next, the display control processing according to this embodiment will be described in detail with reference to Figures 23 to 45. In the display control processing shown in Figure 23, the acquisition unit 11 acquires control data 131 from the PLC 20 (step S21), and the control data processing unit 17 executes control data information extraction processing (step S22).

[0102] In the control data information extraction process, as shown in Figure 24, the control data processing unit 17 identifies the type of control data 131 (step S221). Specifically, the device of the device value indicated by the control data 131 is identified based on the metadata contained in the control data 131, or by querying the PLC 20 or the user. For example, it is identified that the device value is stored in device "Y10" corresponding to the address of the memory 210 of the PLC 20.

[0103] In the example shown in Figure 5, a single control data 131 was represented as containing time-series device values ​​read from multiple devices. However, the control data 131 may also represent a time-series device value read from a single device. Furthermore, time-series device values ​​read from the same device at different time periods may be treated as separate control data 131.

[0104] Returning to Figure 24, following step S221, the control data processing unit 17 identifies the range of each control data 131 (step S222). For example, for a device value that changes as shown in Figure 25, the range from its minimum value of -20000 pulses / ms to its maximum value of 20000 pulses / ms is identified. Similarly, for a device value that changes as illustrated in Figure 26, the range from its minimum value of -10000 pulses / ms to its maximum value of zero pulses / ms is identified. Note that the device values ​​shown in Figures 25 and 26 represent the number of pulses proportional to the speed at the machine end.

[0105] Next, the control data processing unit 17 identifies the number of device values ​​in each control data 131 (step S223). That is, the number of sampling points corresponding to the number of times recorded in the time axis direction is identified. Along with the number of sampling points, the sampling period may also be identified. The number of sampling points and the sampling period are identified based on metadata contained in the control data 131, or by querying the PLC 20 or the user.

[0106] Note that, since the explanation will focus on examples where the number of pulses indicated by the device value directly represents the controlled quantity, the control data information extraction process in Figure 24 is omitted. However, if the controlled quantity can be derived by performing calculations on the device value, a step to derive the controlled quantity may be inserted.

[0107] Next, the control data processing unit 17 stores the information identified in steps S221 to S223 in the storage unit 13 (step S224), and provides the stored information to the display control unit 16 as appropriate. After that, the processing by the display control device 10 returns to the display control processing shown in Figure 23.

[0108] Following step S22, the acquisition unit 11 acquires the image data 132 (step S23), and the image processing unit 14 performs image data information extraction processing (step S24).

[0109] In the image data information extraction process, as shown in Figure 27, the image processing unit 14 identifies the total number of frames (step S241) and the image size (step S242) for each image data 132. For example, for image data 132 showing a moving image generated by shooting for 3 seconds at 1000 fps (frames per second), the total number of frames is identified as 3000. The image size is identified as, for example, an image with a width of 600px and a height of 400px.

[0110] Next, the image processing unit 14 performs image processing (step S243) and identifies the range of each image processing result (step S244). For example, when the workpiece 42 is tracked as an object in the image processing, it is identified that the movement speed in the X-axis direction of the object shown in Figure 28 fluctuates within the range from its minimum value of -1.0 pixels / ms to its maximum value of 1.0 pixels / ms. It is also identified that the movement speed in the Y-axis direction of the object illustrated in Figure 29 fluctuates within the range from its minimum value of zero pixels / ms to its maximum value of 0.5 pixels / ms.

[0111] Next, the image processing unit 14 stores the information identified in steps S241, 242, and 244 in the storage unit 13 (step S245), and provides the stored information to the display control unit 16 as appropriate. After that, the display control device 10 returns to the display control process shown in Figure 23.

[0112] Returning to Figure 23, following step S24, the reception unit 12 receives the conversion parameter 136, and the control data processing unit 17 converts the unit of the control quantity to pixel units (step S25). For example, the screen shown in Figure 30 is presented to the user, and the conversion parameter 136 is entered into the input field 521 on this screen.

[0113] In the example shown in Figure 30, the "Unit Setting" item in input field 521 is set to convert "pulse" units to "pixel" units. Also, since the "Automatic Setting" item is set to OFF, the values ​​entered by the user are set for the other items, and these set values ​​are accepted as conversion parameters 136.

[0114] As illustrated in Figure 31, when the "Automatic Setting" item is set to ON, the control data processing unit 17 calculates the conversion parameters 136 using a predetermined internal algorithm. When the internal algorithm calculates the conversion parameters 136, the reception unit 12 receives the setting to turn ON the "Automatic Setting" item from the user as the conversion parameter 136, and the other conversion parameters 136 are calculated and set by the control data processing unit 17.

[0115] For example, the control data processing unit 17 performs normalization processing using Min-Max Scale or Min-Max Normalization. According to this method, the range of values ​​is scaled to a certain range. Specifically, ND[i] is set as the i-th value of the normalized data to be derived, CD[i] is set as the i-th value of the time-series control variable, MaxCD is set as the maximum value of the control variable, and MinCD is set as the minimum value of the control variable, and ND[i] is calculated by the following equation (1).

[0116] ND[i]=(CD[i]-MinCD) / (MaxCD-MinCD)...(1)

[0117] As a result, the changes in the controlled variables shown in Figures 25 and 26 are normalized and converted into controlled variables that change within the range of zero to one, as shown in Figures 32 and 33.

[0118] Next, the control data processing unit 17 integrates the coordinate system of the controlled variable and the coordinate system of the image. For example, as shown in Figure 34, the direction of the X axis of the image coordinate system and the direction of the X axis of the machine end coordinate system may coincide, while the direction of the Y axis of the image coordinate system and the direction of the Y axis of the machine end coordinate system may be opposite. In such cases, the Y axis direction of the controlled variable is reversed to make the directions coincide. Specifically, YNDr[i] is used as normalized data with the Y axis direction reversed, and YND[i] is used as normalized data to be reversed in the Y axis direction, and YNDr[i] is used as normalized data with the Y axis direction reversed, and YNDr[i] is calculated using the following equation (2).

[0119] YNDr[i]=-1×YND[i]...(2)

[0120] Next, the control data processing unit 17 scales the range of the control variable by fitting it within the range of the image processing result as illustrated in Figures 28 and 29. Specifically, SD[i] is the scaled data to be derived, ND[i] is the normalized data to be scaled, MaxIPD is the maximum value of the image processing result, and MinIPD is the minimum value of the image processing result, and SD[i] is calculated using the following equation (3).

[0121] SD[i]=ND[i]×((MaxIPD-MinIPD)+MinIPD)...(3)

[0122] As a result, the time-series control variables exemplified in Figures 32 and 35 are transformed as exemplified in Figures 36 and 37. Furthermore, comparing this with the image processing results exemplified in Figures 28 and 29, it can be seen that the image processing results progress roughly in the same way as the control variables, as exemplified in Figures 38 and 39, indicating that the processing on the workpiece 42 is proceeding to a certain extent as intended. This converts the units of the control variables to a size suitable for displaying them superimposed on the image, and the preparation for displaying the figures representing the control variables is complete.

[0123] Returning to Figure 23, following step S25, the reception unit 12 receives the display parameter 137 and displays the graphic superimposed on the image according to the user's operation (step S26). Specifically, the display parameter 137 entered in the input field 522 of the "graphic display setting" exemplified in Figure 40 is received by the reception unit 12.

[0124] In the "Display Range" field of input field 522, the drawing range of the shape to be displayed overlaid on the image is set. In the example in Figure 40, the shape is set to be drawn within the range of 0 pixels to 800 pixels in the X-axis direction and within the range of 0 pixels to 600 pixels in the Y-axis direction. The origins in the X-axis and Y-axis directions are the upper left endpoint of the image display area 523, with the X-axis direction corresponding to the right and the Y-axis direction corresponding to the downward direction.

[0125] In the "Overflow Range" field of input field 522, the range in which the shape is allowed to overflow relative to the "Display Range" is set. In the example in Figure 40, an overflow range of 100 pixels is set in both the X-axis and Y-axis directions, meaning that the shape can be drawn within the range of -100 pixels to 900 pixels in the X-axis direction and -100 pixels to 700 pixels in the Y-axis direction.

[0126] The "Shape Display Magnification" item in input field 522 is set to make it easier for the user to see the shape if its size is inappropriate. In the example in Figure 40, the shape is set to be drawn at 3 times its original size. Note that the "Shape Display Magnification" setting applies to shapes representing velocity and acceleration, but not to shapes representing position.

[0127] In the input fields 522, the items "Shape 1," "Shape 2," and "Shape 3" specify the type of shape to be drawn for each controlled quantity or image processing result. In the example in Figure 40, it is set to draw a thick arrow for the controlled quantity "Control Speed," a white arrow for the image processing result "Movement Speed," and a circular mark for the controlled quantity "Current Position."

[0128] When a display parameter 137 is entered into the input field 522, the display control unit 15 overlays a figure representing the control amount at the same time onto the image contained in the image data 132 in the image display area 523, and displays it as shown in Figure 40. In the example in Figure 40, the length and direction of each arrow represent the control amount or image processing result in the X-axis and Y-axis directions. The color scheme, pattern, and size of the arrows follow the settings made by the user in the input field 522. The circular mark at the starting point of the arrow represents the current position of the workpiece 42. The current position may be obtained as an image processing result.

[0129] The shapes displayed in the image display area 523 are changed as appropriate according to the input in the input field 522. For example, as shown in Figure 41, a thick arrow representing the control acceleration, which is a control quantity, may be drawn, and a white arrow representing the motion acceleration, which is the image processing result at the same time, may also be drawn. These accelerations can be obtained by differentiating the velocity as shown in Figure 40. However, if there is a device value that directly represents the acceleration, the control data processing unit 17 may obtain the acceleration from this device value and obtain the velocity by integrating that device value.

[0130] Furthermore, the combination of displayed shapes can be arbitrarily changed by modifying the input content in input field 522. For example, an arrow indicating the control velocity as a control quantity and an arrow indicating the control acceleration at the same time may be drawn as shown in Figure 42, or an arrow indicating the movement speed as an image processing result and an arrow indicating the movement acceleration at the same time may be drawn as shown in Figure 43.

[0131] Furthermore, the display screen shown in Figure 30 includes an input field 524 into which display parameters 137 different from those in input field 522 are entered. When the "Name," "Value," and "Unit" checkboxes in this input field 524 are checked, the name, value, and unit of the value corresponding to the drawn shape are displayed, as shown in Figure 44. For example, in the image display area 523 of Figure 44, the strings "Control Speed," "5000," and "pulse / ms" are displayed near the thick arrow.

[0132] Furthermore, when the "Grid Lines" check box in the input field 524 is checked and the grid width is entered in the box adjacent to that check box, grid lines are displayed in the image display area 523 as dashed lines. These grid lines are based on image size information extracted from the image data 132.

[0133] When the "Cursor Position" checkbox in the input field 524 is checked, the coordinate values ​​of the cursor in the image display area 523 are displayed. If it is possible to derive a control variable corresponding to these cursor coordinate values, the control variable may be displayed along with the coordinate values. In the example in Figure 44, it is shown that the cursor coordinate values ​​are (140, 50) pixels and the corresponding control variable is (1400, 500).

[0134] When the "Image Processing Result" checkbox in the input field 524 is checked, the result obtained from image processing is displayed in the image display area 523. In the example in Figure 44, the coordinate values ​​(300, 330) of the current position are displayed as the image processing result.

[0135] When the "distance between two points" checkbox in the input field 524 is checked, the user can set any two points on the image display area 523, and the distance between those two points will be displayed. In the example in Figure 44, the distance from the circular mark to the cursor position is displayed. If it is possible to derive a control quantity corresponding to this distance, the distance in units of the control quantity may be displayed along with the distance in units of pixels.

[0136] Furthermore, the display screen shown in Figure 30 includes an input field 525 for playing back the image shown in the image display area 523 as a moving image. As shown in Figure 44, buttons 621, 622, 623, and 624 for "playback," "step playback," "slow motion playback," and "playback speed setting" of the moving image are arranged in this input field 525. When the "playback" button 621 is pressed, the display control unit 15 plays back the graphics generated from the control data 131 and the images contained in the image data 132 in the image display area 523 while synchronizing them. When the "slow motion playback" button 623 is pressed, the display control unit 15 plays back the moving image at the speed selected using the "playback speed setting" button 624. In addition, when the user operates the knob of the playback bar in the input field 525, the image and graphics corresponding to the time of that knob are drawn in the image display area 523.

[0137] The input field 525 also includes a section for displaying the video number as an identifier for the image data 132, a check box for registering to the comparison list, and a button 625 for starting the comparison of the video images registered to the comparison list. When button 625 is pressed, a sub-window 526 named "Multiple Data Comparison" is displayed, as shown in Figure 45. This sub-window 526 displays the video images registered to the comparison list in parallel. The sub-window 526 also displays tables for comparing information related to the control data 131 and information related to the image data 132, respectively. This makes it easy to compare information related to each of the multiple video images, as well as information related to device values ​​collected in synchronization with the video images. The information related to the video images may include information extracted by the image data extraction process and the results of image processing. The information related to device values ​​may include information extracted by the control data extraction process and control quantities.

[0138] The reception unit 12, which accepts registration to the comparison list, corresponds to an example of reception means that accepts the selection of two or more video data from multiple video data. The display control unit 15 corresponds to an example of display control means that displays superimposed images including images and graphics related to the two or more selected video data in parallel.

[0139] As described above, the display control unit 15 superimposes a figure having a position or size representing the controlled quantity onto an image of the device value corresponding to the controlled quantity at the same time, and plays a moving image on the display device. As a result, the user can recognize the change in the controlled quantity by viewing the moving image. Therefore, the matching work using images of processing steps performed at the factory automation site can be made simpler.

[0140] Furthermore, the controlled quantity is the speed or acceleration of the machine end acting on the workpiece 42. Both speed and acceleration are important factors for determining whether the processing steps of the workpiece 42 have been carried out appropriately. For this reason, by superimposing a graphic representing the controlled quantity, which is speed or acceleration, an effective matching operation can be easily performed. Note that the object being photographed may be the machine end.

[0141] Furthermore, the reception unit 12 receives conversion parameters 136 via the user interface, and the display control unit 15 calculates the position or size of the figure from the controlled quantity based on the conversion parameters 136. This allows the user to display a figure at an appropriate position or size as needed. The display control unit 15 may calculate both the position and size of the figure from the controlled quantity. The reception unit 12 is an example of a reception means that receives conversion parameters via the user interface for converting the controlled quantity into at least one of the position and size of the figure in pixel units.

[0142] Furthermore, if a setting to automatically convert the units of the control quantity is accepted, the display control unit 15 calculates the position or size of the figure from the control quantity according to a predetermined procedure. This reduces the effort required of the user to set the conversion parameters 136 in detail. The display control unit 15 may also automatically calculate both the position and size of the figure from the control quantity. The reception unit 12 is an example of a reception means that receives an instruction to automatically convert the control quantity to at least one of the position and size of the figure in units of pixels.

[0143] Furthermore, the display control unit 15 displays a figure representing the result of image processing on the display unit 16, along with a figure representing the controlled quantity. Moreover, this image processing result is the result of detecting physical attributes equivalent to the controlled quantity in the image. That is, a figure representing the velocity of the machine end as the controlled quantity is displayed along with a figure representing the velocity of the object detected in the image processing. Also, a figure representing the acceleration of the machine end as the controlled quantity is displayed along with a figure representing the acceleration of the object detected in the image processing. This makes the matching process between the control data 131 and the image data 132 more intuitive.

[0144] Furthermore, the control data processing unit 17 calculates the position or size of a figure by performing data processing such as differential, integral, and statistical processing on the device value or control variable. For example, the speed of the machine end can be derived by differentiating a device value that is proportional to the actual position of the machine end. Also, by integrating a control variable that is proportional to the speed, a new control variable indicating the position of the machine end can be obtained, making it possible to switch the display of the figure. In addition, noise may be removed by applying statistical processing such as a moving average. This makes it possible to display figures corresponding to various physical states. The control data processing unit 17 is an example of a data processing means that calculates at least one of the position and size of a figure from a control value by performing processing including differential, integral, or statistical processing.

[0145] Furthermore, the display content according to this embodiment may be combined with the display content according to Embodiment 1 described above. For example, as shown in Figure 46, a figure representing the controlled quantity may be displayed superimposed on an image, and a graph showing the transition of the result value of image processing may be displayed superimposed on a graph showing the transition of the controlled value. A figure displayed superimposed on an image extracted from a specific timing of a moving image represents the situation at that timing and is effective for matching instantaneous, i.e., microscopic, states. On the other hand, the superimposed display of a graph shows the entire time series and is therefore effective for matching from a macroscopic perspective.

[0146] While embodiments of this disclosure have been described above, this disclosure is not limited to the embodiments described above.

[0147] For example, we have described an example in which a timestamp is added to the device value in the control data 131, and the logging of the device value and the capture of the image are synchronized by an I / O signal, but we are not limited to this. For example, as shown in Figure 47, if a timestamp is also added to the image data 132, the synchronized device value and image can be identified by comparing the timestamp of the control data 131 with the timestamp of the image data 132. Also, as shown in Figure 48, if the frame count "1" in the image data 132 corresponds to the timestamp "1002034" in the control data 131, and synchronization information indicating that the capture interval is 4ms is provided, the synchronized device value and image can be identified based on this synchronization information.

[0148] Furthermore, while we have described an example of displaying the changes in device values ​​and the changes in the image processing results using two corresponding graphs, the method is not limited to this. For example, the synchronized device values ​​and the image results may be displayed in the same column in a table format, as shown in Figures 5, 47, and 48. Also, if the result of processing a single image is represented by a spectrum, the time series may be displayed as a spectrogram in time-frequency representation.

[0149] Furthermore, although an example has been described in which the control data 131 is provided from the PLC 20, it is not limited to this, and may be provided from other industrial PCs, servo amplifiers, machine tools, or other FA devices different from the display control device 10.

[0150] Furthermore, the display control device 10 may be configured without the display unit 16. As shown in Figure 49, the display control unit 15 may cause the information to be displayed on an external display device 16a.

[0151] Furthermore, devices other than the servo amplifier 21 and servo motor 22 according to Embodiment 1, and the motor and actuators 222 and 223 according to Embodiment 2, may be controlled. Examples of such controlled devices include FA equipment such as machine tools, actuators, or robots.

[0152] Furthermore, the communication lines that enable communication between devices may be a network such as a LAN or a wide-area communication network.

[0153] Furthermore, although the explanation has focused on examples where the objects being photographed are workpieces 40 and 42 as controlled objects, the explanation is not limited to these. For example, fluids with irregular shapes, such as grease in a grease production line or sauce in a food manufacturing line, or powders and granules, may be both objects being processed and photographed in the processing step.

[0154] Furthermore, the object being photographed does not necessarily have to be the object being processed. For example, the object may be the controlled equipment itself, a workpiece support, or a component of a belt conveyor.

[0155] The functions of the display control device 10 according to the above-described embodiment can be realized by dedicated hardware or by a normal computer system.

[0156] For example, a device that performs the above-mentioned processing can be configured by distributing program P1 on a computer-readable recording medium such as a flexible disk, CD-ROM (Compact Disk Read-Only Memory), DVD (Digital Versatile Disk), or MO (Magneto-Optical disk), and then installing program P1 on a computer.

[0157] Alternatively, program P1 may be stored on a disk drive of a server device on a communication network such as the Internet, and then downloaded to a computer, for example, by superimposing it onto a carrier wave.

[0158] Furthermore, the above-described process can also be achieved by launching and executing program P1 while transferring it over a network such as the Internet.

[0159] Furthermore, the above-described process can also be achieved by having all or part of program P1 run on a server device, and by having a computer send and receive information related to that process via a communication network while executing program P1.

[0160] Furthermore, if the above-mentioned functions are implemented by the OS (Operating System) or through collaboration between the OS and the application, only the parts other than the OS may be stored and distributed on a medium, or they may be downloaded to a computer.

[0161] Furthermore, the means for realizing the functions of the display control device 10 are not limited to software; some or all of them may be realized by dedicated hardware or circuits.

[0162] This disclosure allows for various embodiments and modifications without departing from the broad spirit and scope of this disclosure. Furthermore, the embodiments described above are for illustrative purposes only and do not limit the scope of this disclosure. In other words, the scope of this disclosure is indicated by the claims, not by the embodiments. Various modifications made within the scope of the claims and the equivalent significance of the disclosure are considered to be within the scope of this disclosure.

[0163] This disclosure is suitable for improving the efficiency of maintenance work at factory automation (FA) sites.

[0164] 10 Display control device, 11 Acquisition unit, 12 Reception unit, 13 Storage unit, 14 Image processing unit, 15 Display control unit, 16 Display unit, 16a Display device, 17 Control data processing unit, 20 PLC, 21 Servo amplifier, 22 Servo motor, 23 Arm, 30-32 Camera, 40, 40a, 42 Workpiece, 41, 41a Mark, 50, 54, 58, 513, 521, 522, 524, 525 Input field, 51, 52, 57, 501, 505, 611-613 List, 53, 56, 61-64, 502-504, 506, 621-625, 701, 702, 712, 713, 715 Button, 55 Result field, 71, 520, 523 Image display area, 72, 530 Graph display area, 81, 531, 703 Line, 101 Processor, 102 Main memory, 103 Auxiliary memory, 104 Input unit, 105 Output unit, 106 Communication unit, 107 Internal bus, 131 Control data, 132 Image data, 133 Image processing parameters, 134 Conversion parameters, 135 Range parameters, 136 Conversion parameters, 137 Display parameters, 201, 202, 301 Communication lines, 210 Memory, 211 Device values, 220 Controlled equipment, 221 Support base, 222, 223 Actuator, 511, 512 Area, 521, 601, 602, 721 Check box, 522, 711 Bar, 526 Subwindow, 714 Flag, 1000 Support system, P1 Program.

Claims

1. A display control device comprising: acquisition means for acquiring time-series control data showing control values ​​used for controlling a controlled device by a control device in relation to time, and moving image data showing moving images of objects related to the control; and display control means for causing a display device to reproduce the moving image by superimposing a figure having at least one of position and size representing a control quantity corresponding to the control value onto an image constituting the moving image, which is an image associated with the time of the control value corresponding to the control quantity.

2. The display control device according to claim 1, wherein the controlled quantity is the speed or acceleration of the object or a mechanical end acting on the object.

3. The display control device according to claim 1 or 2, further comprising: a receiving means for receiving, via a user interface, a conversion parameter for converting the control quantity into at least one of the position and size of the figure in pixel units, wherein the display control means calculates at least one of the position and size of the figure from the control quantity based on the conversion parameter.

4. A display control device according to claim 1 or 2, further comprising: a receiving means for receiving an instruction to automatically convert the control quantity into at least one of the position and size of the figure in pixel units, wherein the display control means calculates at least one of the position and size of the figure from the control quantity according to a predetermined procedure when the instruction is received.

5. A display control device according to any one of claims 1 to 4, further comprising data processing means for calculating at least one of the position and size of the figure from the control value by performing a process including differential processing, integral processing, or statistical processing.

6. A display control device according to claim 1 or 2, further comprising: a receiving means for receiving input via a user interface, wherein the acquisition means acquires a plurality of the motion image data, the receiving means accepts the selection of two or more motion image data from the plurality of motion image data, and the display control means displays a superimposed image including the images and figures relating to the two or more selected motion image data side by side.

7. A display control device according to any one of claims 1 to 6, further comprising: an image processing means for processing the image constituting the moving image, wherein the display control means causes the display device to display on the display device a figure representing the processing result of the image by the image processing means superimposed on the image together with the figure.

8. The display control device according to claim 7, wherein the result of the image processing is the result of detecting the same physical attribute as the controlled quantity in the image.

9. The display control device according to claim 1 or 2, further comprising: a receiving means for receiving the setting of display parameters relating to the display of the figure, wherein the display control means causes the figure to be displayed on the display device in accordance with the setting of the display parameters.

10. A display control method comprising: an acquisition means acquiring time-series control data showing control values ​​used for controlling a controlled device by a control device in relation to time, and moving image data showing moving images of objects related to the control; and a display control means superimposing a figure having at least one of position and size representing a control quantity corresponding to the control value onto an image constituting the moving image, which is an image associated with the time of the control value corresponding to the control quantity, and causing a display device to play the moving image.

11. A program for causing a computer to function as an acquisition means for acquiring time-series control data showing control values ​​used in controlling controlled devices by a control device in relation to time, and moving image data showing moving images of objects related to the control; and a display control means for causing a display device to play the moving image by superimposing a figure having at least one of position and size representing a control quantity corresponding to the control value onto an image constituting the moving image, which is an image associated with the time of the control value corresponding to the control quantity.

Images