Object work system and determination method

Abstract

This object work system comprises: an acquisition unit that acquires, from an imaging unit, a captured image of a conveyance member captured by the imaging unit; a determination unit that determines a state of the conveyance member on the basis of the acquired captured image; and a notification unit that notifies a result determined by the determination unit. The determination unit determines, from the captured image, one or more defective states from among at least two of a repair state in which repair of the conveyance member is required, a dirty state in which dirt adheres to the conveyance member, and a replacement state in which replacement of the conveyance member is required. Thus, energy saving is realized, contributing to CO2 reduction by energy saving. This also contributes to the realization of a smart factory.

Description

Object working system and determination method

[0001] This specification discloses an object working system and a determination method.

[0002] Conventionally, in an electronic component mounting apparatus having a mounting head, there has been proposed a substrate transfer mechanism that supports and transfers a substrate by a transfer belt as a transfer member, and a substrate recognition camera or a height sensor provided in the mounting head for recognizing the surface state of the transfer belt, which determines the deterioration state of the transfer belt from the recognized surface state of the transfer belt (see, for example, Patent Document 1). This apparatus is said to be able to monitor the wear state of the transfer belt while it is mounted on the electronic component mounting apparatus.

[0003] Japanese Patent Application Laid-Open No. 2017-117888

[0004] However, in the above-mentioned Patent Document 1, although the deterioration state of the transfer member is determined from information such as an image captured by a camera or a height sensor, in terms of dealing with problems, it is still not sufficient, and it has been required to deal with problems as early as possible before the occurrence of problems.

[0005] The present disclosure has been made in view of such problems, and the main object is to provide an object working system and a determination method capable of dealing with problems as early as possible.

[0006] The object working system and determination method disclosed in this specification have adopted the following means to achieve the above main object.

[0007] The object handling system of this disclosure is an object handling system having an assembly-related device that performs assembly-related processing related to the assembly process of mounting components onto an object that has been transported by a transport device equipped with a transport member for placing and transporting an object, and comprises an acquisition unit that acquires an image of the transport member captured by an imaging unit from an imaging unit, a determination unit that determines the state of the transport member based on the acquired image, and a notification unit that notifies the result determined by the determination unit, wherein the determination unit determines from the image of the transport member at least two to one or more defective states from among a repair state in which repair is necessary for the transport member, a dirty state in which dirt is attached to the transport member, and a replacement state in which replacement of the transport member is necessary.

[0008] This object handling system uses captured images of the transported material to determine its condition and notifies the system of the determination, allowing for early intervention of potential malfunctions in the transport equipment. This results in energy savings and reduced CO2 emissions. ２ This contributes to reduction. Furthermore, it contributes to the realization of smart factories.

[0009] A schematic diagram showing an example of the implementation system 10. A schematic diagram showing an example of the implementation device 13. A schematic diagram showing an example of the transport device 40. A diagram showing an example of image recording information 24 stored in the storage unit 22. A diagram showing an example of captured images 25a to 25d. A flowchart showing an example of the state estimation model construction processing routine. A flowchart showing an example of the implementation processing routine. A flowchart showing an example of the state determination processing routine. A diagram showing an example of the information display screen 70.

[0010] This embodiment will be described below with reference to the drawings. Figure 1 is a schematic diagram showing an example of the mounting system 10. Figure 2 is a schematic diagram showing an example of the mounting device 13. Figure 3 is a schematic diagram showing an example of the transport device 40, where Figure 3A is a perspective view and Figure 3B is a side view cross-sectional view. Figure 4 is an explanatory diagram showing an example of image recording information 24 stored in the storage unit 22. Figure 5 is an explanatory diagram showing an example of information in the captured image 25, where Figure 5A is an example of captured image 25a showing the conveyor belt 42 in a normal state, Figure 5B is an example of captured image 25b showing a partially damaged state, Figure 5C is an example of captured image 25c showing a dirty state, and Figure 5D is an example of captured image 25d showing a partially exposed state. In this embodiment, the left-right direction (X-axis), front-back direction (Y-axis), and up-down direction (Z-axis) are as shown in Figures 1 to 3.

[0011] The mounting system 10 is configured as an object work system having a production line in which mounting devices 13, which process components P onto a substrate S as an object, are arranged in the transport direction D of the substrate S. Here, the object to be mounted is described as a substrate S, but it is not particularly limited as long as it mounts components P, and may be a three-dimensional base material. This mounting system 10 is configured to be able to exchange information with a control unit 60 via a network 19. The control unit 60 is connected to mounting systems 10, as well as mounting systems 10B, 10C, etc., via the network 19 so that information can be exchanged. Note that mounting systems 10B and 10C may have different configurations from mounting system 10, but a detailed explanation is omitted as they are assumed to include a configuration similar to mounting system 10.

[0012] As shown in Figure 1, the mounting system 10 includes a printing device 11, a printing inspection device 12, a mounting device 13, a mounting inspection device 14, and a management device 18. The printing device 11 is a device that prints a viscous fluid such as solder paste onto a substrate S. The printing device 11 includes a transport device for transporting the substrate S, as well as a printing head with a squeegee that moves the viscous fluid for printing, which serves as a work unit for working on the substrate S as the target object. The printing device 11 may also be a device that prints adhesives or conductive pastes as the viscous fluid. The printing inspection device 12 is a device that inspects the state of the printed viscous fluid. The printing inspection device 12 includes a transport device for transporting the substrate S, as well as an inspection head with an imaging unit that images the printed material, which serves as a work unit for working on the substrate S. The mounting device 13 is a device that mounts components P onto the substrate S. The mounting inspection device 14 is a device that inspects the state of the components P mounted by the mounting device 13. The mounting inspection device 14 includes a transport device for transporting the substrate S, as well as an inspection head with an imaging unit for imaging the substrate S, which serves as a work unit for performing operations on the substrate S. The mounting device 13 may also be a mounting-inspection device that has the functions of the mounting inspection device 14. The mounting system 10 may also include a separate transport device 40. The management device 18 is a computer that manages information on each device of the mounting system 10. The printing device 11, printing inspection device 12, mounting device 13, mounting inspection device 14, and management device 18 are mounting-related devices that perform mounting-related processing related to the mounting process. The mounting-related devices include a transport device for transporting the substrate S, but here we will mainly describe the transport device 40 of the mounting device 13.

[0013] The mounting device 13 is a device that performs a mounting process to mount components P onto a substrate S transported by the transport device 40. The mounting device 13 takes components P from the component supply unit 27 that supplies the components P and mounts them onto the substrate S, which is the target object before or after the printing process. As shown in Figure 2, the mounting device 13 comprises a control device 20, a component supply unit 27, a mounting imaging unit 29, a mounting unit 30, an operation panel 36, and a transport device 40.

[0014] The control device 20 is configured as a microprocessor centered on the control unit 21 and controls the entire device. This control device 20 includes a storage unit 22 for storing various data. The control device 20 has the function of controlling the entire device of the mounting device 13. The control device 20 outputs control signals to each unit of the mounting device 13 and inputs signals from each of these units. The control unit 21, as will be described in detail later, includes an acquisition unit that acquires an image 25 of the conveyor belt 42 as a transporting member, captured by the object imaging unit 34, a determination unit that determines the state of the transporting member based on the acquired image, and a function that notifies the determination result on the operation panel 36 which serves as a notification unit. The control unit 21 may determine from the image that at least two to one malfunction state is among the following: a repair state in which the conveyor belt 42 needs repair, a soiled state in which dirt is attached to the conveyor belt 42, and a replacement state in which the conveyor belt 42 needs to be replaced. Repair and replacement conditions include, for example, a damaged state where a part of the conveyor belt 42 is broken, or an exposed state where internal components inside the conveyor belt 42 are exposed. The control unit 21 also applies the acquired image 25 to a state estimation model 26 constructed by machine learning to determine the malfunction state of the conveyor belt 42. A "malfunction state" refers to a state that is not in good condition, but it may be a state in which the transport process can be performed and is prior to an abnormal state (error state) that hinders the transport process.

[0015] As shown in Figures 2 and 4, the memory unit 22 stores mounting condition information 23, image recording information 24, a state estimation model 26, and the like. The mounting condition information 23 includes information such as the mounting order in which components P are mounted onto the substrate S, the sampling position of components P, the placement position, and the type of sampling member 33 from which components P can be collected. The image recording information 24 includes information on captured images 25 used when determining the state of the transport device 40. As the mounting process by the mounting unit 30 progresses, new captured image information 25 is added to the image recording information 24. The image recording information 24 includes captured images 25 associated with the elapsed time. The captured images 25 include, for example, captured images 25a to 25d, as shown in Figure 5. Captured image 25a is an example of an image of the conveyor belt 42 in a normal state, as shown in Figure 5A. Captured image 25b is an example of an image showing a damaged section B, which is a damaged state in which a part of the conveyor belt 42 is damaged, as shown in Figure 5B. Image 25c is an example of an image showing a dirty area U, where dirt is attached to a part of the conveyor belt 42, as shown in Figure 5C. Image 25d is an example of an image showing an exposed area E, where an internal component located inside the conveyor belt 42 is exposed, as shown in Figure 5D. This image recording information 24 also includes status information regarding malfunctions that have occurred in the conveying device 40. This status information may be, for example, input by an operator after confirming the status of the conveying device 40, or it may be an estimated value by the control unit 21. This status information may include, for example, one or more of the above-mentioned damaged state, dirty state, and exposed state.

[0016] The state estimation model 26 is constructed using machine learning based on the information from the captured image 25. The state estimation model 26 is a learning model that determines the state of the transport device 40 using the information from the captured image 25, which captures the conveyor belt 42 of the transport device 40, as explanatory variables. Examples of machine learning methods include supervised learning, unsupervised learning, reinforcement learning, deep reinforcement learning, and semi-supervised learning, of which supervised learning and semi-supervised learning are preferred. Examples of machine learning methods include support vector machines (SVM), logistic regression, random forests (RF), neural networks (NN), feedforward neural networks (FFNN), recurrent neural networks (RNN), convolutional neural networks (CNN), naive Bayes, principal component analysis, k-nearest neighbors, k-means, and generative adversarial networks (GAN). The machine learning method should be appropriately selected from the above depending on the captured image 25 and the content to be determined. This state estimation model 26 may be constructed based on captured images 25 taken of the conveyor belts 42 of multiple transport devices 40. Here, "multiple transport devices 40" may refer to multiple units of a single type of mounting-related device (for example, only the mounting device 13), or to multiple units of multiple types of mounting-related devices (for example, the printing device 11, the printing inspection device 12, the mounting device 13, and the mounting inspection device 14). Having a variety of training data for machine learning is preferable because it improves the accuracy of determining the state of the conveyor belts 42.

[0017] The state estimation model 26 may, for example, be constructed using, in the initial stage, captured images 25 obtained using a transport device 40 that has been experimentally created in a deteriorated state, and state information including the malfunction at that time, as training data. Alternatively, the state estimation model 26 may be updated thereafter using image recording information 24 obtained when the transport device 40 is used by a user. Examples of deteriorated states include, for example, one of the following on the conveyor belt 42: a damaged part B, a dirty part U, and an exposed part E, and combinations of two or more of these states. The state estimation model 26 is constructed using training data that includes information on each factor in this deteriorated state and good / bad states. The state estimation model 26 is constructed using a number of training data that satisfies a predetermined acceptable accuracy when used to determine the state of the transport device 40. The state estimation model 26 is constructed using the captured images 25 described above, and is configured to identify what kind of malfunction state is occurring when the captured images 25 of the transport device 40 are given.

[0018] The parts supply unit 27 has multiple feeders 28 equipped with reels and tray units, and is detachably attached to the front side of the mounting device 13. Tape, which serves as a holding member, is wound around each reel, and multiple parts P are held along the longitudinal direction of the tape. This holding member is unwound from the reel toward the rear, and with the parts P exposed, it is fed by the feeder 28 to the picking position where it will be picked up by the picking member 33. The mounting imaging unit 29 is a unit that images the mounting unit 30 side, as shown in Figure 2. The mounting imaging unit 29 is a parts camera that captures images of one or more parts P picked up by the mounting head 32. The mounting imaging unit 29 includes an illumination unit that irradiates light onto the mounting unit 30 side and an image sensor that converts the input light into an electrical signal. The mounting imaging unit 29 captures an image of the mounting head 32 holding the parts P as it passes above the mounting imaging unit 29, and outputs the captured image data to the control device 20. The control device 20 processes the captured image to inspect whether the shape and location of part P are normal, and to detect the amount of displacement such as the position and rotation of part P at the time of collection.

[0019] The mounting unit 30 is configured as a work unit that performs work on the substrate S, which is the object. The mounting unit 30 is a unit that picks up components P from the component supply unit 27 using a picking member 33 and places them on the substrate S fixed to the transport device 40. The mounting unit 30 includes a head moving unit 31, a mounting head 32, a picking member 33, an object imaging unit 34, and a lifting unit 35. The head moving unit 31 includes a slider that moves in the XY direction guided by a guide rail and a motor that drives the slider. The mounting head 32 is detachably mounted on the slider and moves in the XY direction by the head moving unit 31. One or more picking members 33 are detachably mounted on the lower side of the mounting head 32. The picking member 33 may be a suction nozzle that picks up components P using negative pressure, or a mechanical chuck that grips components P. The object imaging unit 34 is a unit that images the substrate S, which is the object. The object imaging unit 34 is a mark camera positioned on the underside of the mounting head 32, which images the substrate S and marks formed on the substrate S from above. The object imaging unit 34 moves in the XY direction as the mounting head 32 moves. The object imaging unit 34 is configured to image the conveyor belt 42 in addition to the substrate S. The lifting unit 35 is a unit that moves up and down a cylinder to which the sampling member 33 is attached. The lifting unit 35 is a mechanism that moves linearly along the Z axis, and may be a ball screw mechanism or a linear motor.

[0020] The control panel 36 is a unit that receives input from the operator and displays information to the operator. This control panel 36 is located on the front of the mounting device 13 and includes a display unit, which is a display, and an operation unit that has both a touch panel and buttons.

[0021] The transport device 40 is a unit that loads, transports, fixes at the mounting position, and unloads substrates S along a transport path extending in the left-right direction (X-axis direction). As shown in Figure 3, the transport device 40 comprises a side frame 41, a conveyor belt 42, a drive unit 43, detection units 44 and 45, a support base 47, a lifting unit 48, and a support plate 49. The transport device 40 also includes a clamping device for fixing the substrates S. The side frame 41 is a pair of support members arranged along the transport direction D at a predetermined distance in the Y-axis direction. One of the side frame 41 is fixed to the end of the support base 47, and the other moves along a guide formed on the support base 47, allowing the transport width to be changed. The conveyor belt 42 is provided on each of the pair of side frames 41 and is a transport member that carries objects on it. The conveyor belt 42 is stretched between a drive shaft and a driven shaft and is driven to circumferentially along the transport direction D by the rotation of the drive shaft. The drive unit 43 is a motor connected to the drive shaft and rotates the drive shaft. The drive unit 43 may be, for example, a stepping motor capable of driving in both forward and reverse rotation directions. The detection units 44 and 45 are sensors provided on the transport path of the transport device 40, as shown in Figure 3B, and are used to detect objects. The detection units 44 and 45 may be, for example, transmissive optical sensors comprising a light-emitting unit and a light-receiving unit provided at opposing positions on either side of the transport path of the substrate S. The detection units 44 and 45 may also be reflective optical sensors. The detection unit 44 is provided on the entrance side of the transport path, and the detection unit 45 is provided in the central part between the entrance and exit of the transport path. The control device 20 acquires the movement and stopping position of the substrate S based on the detection signals from the detection units 44 and 45. The support base 47 is a base that supports the components of the transport device 40 from below. The lifting mechanism 48 is mounted on the support base 47 and is a mechanism for raising and lowering the support plate 49. The support plate 49 is a support member that supports the substrate S from below, and for example, backup pins are arranged on it.

[0022] The central control unit 60 is configured as a server for a central control system that manages multiple implementation systems, including implementation systems 10, 10B, 10C, etc. The central control unit 60 may be owned by the developer or provider of the implementation systems 10, 10B, 10C, etc. The central control unit 60 transmits signals to a management device 18, etc., provided by the implementation system 10, and receives signals from this management device 18, etc. As shown in Figure 1, the central control unit 60 comprises a control unit 61, a storage unit 62, a display device 67, an input device 68, and a communication unit 69. The control unit 61 is configured as a microprocessor and controls the entire device. The storage unit 62 is a storage medium that stores various data and programs. The storage unit 62 stores implementation condition information 63, image recording information 64, and a state estimation model 66, which include information similar to the implementation condition information 23, image recording information 24, and state estimation model 26 stored in the storage unit 22. The image recording information 64 may include an image 65 of the conveyor belt 42 and information about the state of the conveyor belt 42 at that time. Alternatively, the image recording information 64 may include an image 65 obtained during the manufacturing of the substrate S, acquired from the mounting systems 10, 10B, 10C, etc., and information about its state. The state estimation model 66 may be pre-constructed using the image recording information 64 before being used in the mounting system 10. The information stored in the storage unit 62 will not be explained here, as the explanation of the information stored in the storage unit 22 is provided. The control unit 61 creates mounting condition information 63, acquires and saves the image recording information 64 from the mounting system 10, and executes a process to construct the initial state of the state estimation model 66. The display device 67 is a liquid crystal screen that displays various information. The input device 68 includes a keyboard and mouse, etc., for the supervisor to input various commands. The communication unit 69 is an interface for exchanging information with external devices such as the management device 18 of the mounting system 10.

[0023] Next, the operation of the implementation system 10 of this embodiment, as configured in this way, will be explained, starting with the process of constructing the state estimation model 26. The state estimation model 26 may be constructed by the control unit 60, the management device 18, or the control device 20. Here, the case in which the control unit 60 newly constructs the state estimation model 66 will be explained as an example. Figure 6 is a flowchart of an example of a state estimation model construction processing routine. This routine is stored in the storage unit 62 and executed by instruction from the user using the control unit 60. This routine may be executed when the state estimation model 26 is pre-constructed before the implementation system 10 is put into operation, or it may be executed when the state estimation model 26 is updated and constructed after the implementation system 10 has been put into operation and data of the captured images 65 has been accumulated in the production process.

[0024] When this routine is started, the control unit 61 of the central unit 60 first obtains the captured image 25 and the state information at the time the captured image 25 was acquired from the image recording information 64 as training data to be used in the state estimation model 66 to be constructed (S100). Next, the control unit 61 has the acquired information incorporated into the state estimation model 66 (S110) and updates the state estimation model 66 that has incorporated the captured image 65 (S120). Subsequently, the control unit 61 determines whether a standard number of captured images 65 have been acquired (S130), and if a standard number of captured images 65 have not been acquired, it executes the processing from S100 onwards. Here, the "standard number" may be, for example, a value that indicates a predetermined allowable accuracy (such as 80% or more or 90% or more) when determining the state of the transport device 40 using the state estimation model 66, and is set to the value obtained by experience. When the standard number of captured images 65 have been acquired in S130, the control unit 61 outputs the state estimation model 66 constructed thereto to the implementation system that will use it (S140), and terminates this routine. The implementation system 10 stores the acquired state estimation model 66 as state estimation model 26 in the storage unit 22. The implementation system 10 uses the state estimation model 26 to determine if the configuration of the transport device 40 is deviating from a good state.

[0025] Next, the process of placing components P onto the substrate S using the mounting device 13 will be described. Figure 7 is a flowchart of an example of a mounting process routine executed by the control device 20 of the mounting device 13. This routine is stored in the storage unit 22 of the mounting device 13 and executed by the control unit 21 of the control device 20 after the mounting system 10 is started. When this routine is started, the control unit 21 first reads and acquires the mounting condition information 23 for the substrate S to be produced this time from the storage unit 22 (S200). The control unit 21 may also acquire the mounting condition information 23 from the management device 18.

[0026] Next, the control unit 21 determines whether or not it is time to acquire the captured image 25 (S210). This acquisition timing may be a periodic timing, such as exceeding a predetermined period (e.g., one day) or a predetermined number of conveyed items (e.g., one thousand sheets). The control unit 21 may change this acquisition timing and the status determination timing (interval) according to the current usage status of the conveyor belt 42. This usage status may include, for example, one or more of the following: the usage time of the conveyor belt 42, the weight of the substrate S as the object, and the surface roughness of the substrate S as the object. For example, the control unit 21 may change the acquisition timing to a shorter interval as the usage time of the conveyor belt 42 increases, the weight of the substrate S increases, and / or the surface roughness of the substrate S increases. When it is time to acquire the captured image 25, the control unit 21 moves the object imaging unit 34 above the conveyor belt 42, drives the conveyor belt 42 as a transport member, has the object imaging unit 34 take an image, and stores the obtained captured image 25 in the image recording information 24 (S220). The control unit 21 drives the conveyor belt 42 in a circular motion and controls the drive unit 43 to stop at a predetermined transport distance. At this time, the control unit 21 may rotate the conveyor belt 42 by one full rotation, or it may rotate the conveyor belt 42 so that frequently used parts are imaged. The control unit 21 may, for example, rotate the conveyor belt 42 when there are no substrates S on the conveyor belt 42, such as when waiting for substrates or before production starts, and take an image. The control unit 21 may continuously image the circularly driven conveyor belt 42 with uninterrupted still images and acquire the captured image 25. Alternatively, the control unit 21 may capture images of the conveyor belt 42 as a video.

[0027] After S220, or when it is not time to acquire the captured image 25 in 210, the control unit 21 transports the substrate S, stops the substrate S at a predetermined working position based on the signals from the detection units 44 and 45, and fixes the substrate S (S230). Next, the control unit 21 sets the components P to be placed on the substrate S based on the mounting condition information 23 (S240). The control unit 21 sets the components P to be placed in the mounting order, for example, according to the number of sampling members 33 attached to the mounting head 32. Next, the control unit 21 moves the mounting head 32 to the component supply unit 27 and has the sampling members 33 collect the components P set as the target of placement (S250). Next, the control unit 21 moves the mounting head 32 above the mounting imaging unit 29 and has the mounting imaging unit 29 image the components P collected by the mounting head 32 (S260). Next, the control unit 21 acquires a correction value to cancel the amount of displacement in the sampling state based on the captured image and temporarily stores this correction value (S270). Next, the control unit 21 moves the mounting head 32 to the placement position and uses this correction value to correct the misalignment and place the component P on the substrate S (S280).

[0028] After S280, the control unit 21 determines whether there is a component P to be picked up and placed next (S290). If there is a component P to be picked up and placed next, it executes the process from S240 onwards. That is, the control unit 21 sets the component P to be picked up next and places the component P on the substrate S while correcting for misalignment. On the other hand, if there is no component P to be picked up and placed next at S290, the control unit 21 determines whether the production of the substrate S is complete (S300). If the production of the substrate S is not complete, the control unit 21 executes the process from S210 onwards. On the other hand, if the production of the substrate S is completed at S300, this routine is terminated. In this way, the control unit 21 performs the mounting process while periodically acquiring captured images 25 of the conveyor belt 42.

[0029] Next, the process by which the control device 20 determines whether a malfunction has occurred in the transport device 40 will be described. Figure 8 is a flowchart showing an example of a state determination processing routine executed by the control device 20 of the mounting device 13. This routine is stored in the storage unit 22 of the mounting device 13 and is repeatedly executed by the control unit 21 of the control device 20 in parallel with the mounting processing routine. When this routine is started, the control unit 21 first determines whether or not the state determination timing has been reached (S400). The state determination timing may be one or more of the following, for example, the start of production of the substrate S, the elapsed time of a predetermined period, or the production of a predetermined number of substrates S. Currently, if the state determination timing has been reached, the control unit 21 acquires the most recent captured image 25 (S410). The control unit 21 acquires the captured image 25 obtained in S220 described above. Next, the control unit 21 applies the acquired captured image 25 to the state estimation model 26 and uses the state estimation model 26 to determine the state of the conveyor belt 42 of the transport device 40 (S420). The control unit 21 uses the image features of the conveyor belt 42 included in the captured image 25 as explanatory variables to determine whether one or more defects such as damaged parts B, soiled parts U, or exposed parts E have begun to occur on the conveyor belt 42. Next, the control unit 21 determines whether there is a defect state, i.e., an unfavorable state, on the conveyor belt 42 of the transport device 40 (S430). If there is no defect state, or if it is not the timing for determining a defect state in S400, the control unit 21 terminates this routine. On the other hand, if it is determined that there is a defect state, i.e., an unfavorable state, on the conveyor belt 42, the control unit 21 obtains the type of defect state, i.e., damaged parts B, soiled parts U, exposed parts E, and two or more combinations thereof, from this determination result (S440). Alternatively, the control unit 21 may determine one or more defect states from at least two of the following: a repair state where the conveyor belt 42 needs repair, a soiled state where dirt has adhered to the conveyor belt 42, and a replacement state where the conveyor belt 42 needs to be replaced, from the captured image 25. The control unit 21 may determine whether the damaged part B or the exposed part E is in a repair state or a replacement state depending on the extent of the damage. The control unit 21 then outputs information regarding the nature of the malfunction to the notification unit (S450) and terminates this routine.The notification unit may be, for example, an operation panel 36, or a management device 18 or a control unit 60.

[0030] Figure 9 is an explanatory diagram showing an example of an information display screen 70 displayed on the display unit of the operation panel 36 of the mounting device 13. The information display screen 70 includes an information display area 71 and a message display area 72. The information display area 71 is a field where information such as the status of the device is displayed, and it also includes the message display area 72. The message display area 72 is a field where messages to the operator, such as instructions for work on the mounting device 13, are displayed. When the control unit 21 estimates that the conveyor belt 42 of the transport device 40 is in a malfunction state before it becomes an abnormal state that makes it difficult to operate the device, it displays a message on the operation panel 36 indicating that an inspection should be performed, along with information such as the estimated transport device 40 and the nature of the malfunction state. When the operator confirms this message, they can perform work to resolve the malfunction state at a relatively early stage.

[0031] Generally, when judging deterioration, a threshold is set for a certain factor, and superiority or inferiority is determined by whether or not the threshold is exceeded. In this case, the threshold is set closer to the abnormal state due to the accuracy of the judgment, so it was not possible to accurately determine the malfunction state before it reached an error state. This control device 20 uses a state estimation model 26 that has been machine-learned using the image features contained in the captured image 25 as explanatory variables, so it can accurately estimate the malfunction state before it reaches an abnormal state even when using captured image 25 where it is difficult to set a threshold. In particular, the control device 20 machine-learns damaged parts B, dirty parts U, exposed parts E, etc. in combination in the captured image 25, so it can estimate the nature of the malfunction more accurately.

[0032] Next, the process by which the control device 20 updates the state estimation model 26 will be described. The control device 20 executes the state estimation model construction processing routine shown in Figure 6 and updates the state estimation model 26 using the new captured image 25 and the state information, which is information about the defect state, as training data. The state information may include, for example, information about the actual defect state confirmed by the operator (for example, damaged part B occurred). In this way, the control device 20 updates the state estimation model 26 in accordance with the mounting process of the mounting device 13, enabling more appropriate estimation.

[0033] Here, the correspondence between the components of this embodiment and the components of the present disclosure will be clarified. The transport device 40 of this embodiment is an example of a transport device of the present disclosure, the mounting system 10 is an example of an object work system, the control unit 21 is an example of an acquisition unit, a determination unit, or a control unit, the conveyor belt 42 is an example of a transport member, and the mounting unit 30 is an example of a work unit. Furthermore, the damaged part B, the soiled part U, and the exposed part E are examples of the repaired state, the soiled state, and the replacement state, the captured image 25 and the captured image 65 are examples of captured images, and the state estimation model 26 and the state estimation model 66 are examples of state estimation models. Furthermore, the operation panel 36, the management device 18, and the control unit 60 are examples of notification units, the damaged part B, the soiled part U, and the exposed part E are examples of the damaged state, the soiled state, and the exposed state, respectively, the printing device 11, the printing inspection device 12, the mounting device 13, and the mounting inspection device 14 are examples of mounting-related devices, part P is an example of a part, and substrate S is an example of an object. In this embodiment, an example of the determination method of this disclosure is also revealed by explaining the operation of the mounting device 13.

[0034] The mounting system 10 as an object handling system of this embodiment, as described above, has a mounting-related device that performs mounting-related processing for mounting components P onto a substrate S transported by a transport device 40 equipped with a conveyor belt 42 as a transport member for transporting a substrate S as an object. The mounting system 10 acquires an image 25 of the conveyor belt 42 captured by the object imaging unit 34 from the object imaging unit 34, determines the state of the conveyor belt 42 based on the acquired image 25, and performs processing to report the result of the determination. The control unit 21, which acts as both an acquisition unit and a determination unit, determines from the image image at least two to one of the following fault states: a repair state in which the conveyor belt 42 needs repair, a dirty state in which dirt is attached to the conveyor belt 42, and a replacement state in which the conveyor belt 42 needs to be replaced. Furthermore, the control device 20 includes a control unit 21 that uses a state estimation model 26 constructed by machine learning based on the information of the captured images 25 of the conveyor belt 42, acquires the captured images 25 of the conveyor belt 42 from the object imaging unit 34, and applies the acquired captured images 25 to the state estimation model 26 to estimate the state of the conveyor belt 42. In this control device 20, the state of the conveyor belt 42 is determined by applying the captured images 25 to the state estimation model 26 constructed by machine learning based on the information of the captured images 25 of the conveyor belt 42. Therefore, this control device 20 can determine malfunction states that match the state estimation model 26 and, by using the determination results, can address malfunctions that may occur in the conveying device 40 as early as possible.

[0035] Furthermore, the control unit 21 uses a state estimation model 26 to determine from the captured image 25 one or more of the following malfunction states: a damaged state with a damaged portion B where a part of the conveyor belt 42 is damaged, a soiled state with a soiled portion U attached to the conveyor belt 42, and an exposed state with an exposed portion E where an internal member inside the conveyor belt 42 is exposed. This control device 20 can address any malfunction that may occur in the conveyor belt 42, such as a damaged state, a soiled state, or an exposed state, as early as possible. Moreover, when a malfunction state is determined in the conveyor belt 42, the control unit 21 outputs the determined malfunction state to the operation panel 36, management device 18, and control unit 60, which act as notification units. By using the outputted malfunction state, this control device 20 can address any malfunction that may occur in the conveying device 40 as early as possible.

[0036] Furthermore, the control unit 21 acquires captured images 25 of the conveyor belt 42 of the transport device 40 in the implementation system 10, and state information regarding defects in the conveyor belt 42 determined by the operator, as training data, and updates the state estimation model 26. This control device 20 can address potential defects in the transport device 40 with greater accuracy by updating the state estimation model 26 using captured images 25 taken during use in the implementation system 10. Moreover, the state estimation model 26 is constructed based on captured images 25 taken of the conveyor belts 42 of multiple transport devices 40. This control device 20 can address potential defects in the transport device 40 with greater accuracy by constructing the state estimation model 26 using captured images 25 taken of defects that occurred in various transport devices 40.

[0037] Furthermore, the control unit 21 may acquire images of the conveyor belt 42 captured by the object imaging unit 34, which images the substrate S transported by the transport device 40. In this control device 20, the imaging unit that captures both the substrate S and the conveyor belt 42 is shared, which simplifies the device configuration and allows for earlier response to potential malfunctions in the transport device 40. In addition, the control unit 21 periodically acquires captured images 25 from the object imaging unit 34 and periodically determines the state of the conveyor belt 42 using the state estimation model 26. In this control device 20, by periodically acquiring captured images 25, potential malfunctions in the transport device 40 can be determined more reliably.

[0038] Furthermore, the control unit 21 can determine the state of the conveyor belt 42 using the captured image 25 of the conveyor belt 42 in a driven state, thereby determining the overall state of the conveyor belt 42. In addition, the control unit 21 acquires an captured image 25 from the object imaging unit 34, which is at least one of a video or still image captured by the object imaging unit 34, so it can determine the state of the conveyor belt 42 using either a video or a still image. Furthermore, the control unit 21 acquires an captured image 25 of the conveyor belt 42 without the substrate S as the object, so it can determine the overall state of the conveyor belt 42. Moreover, the control unit 21 determines the malfunction state of the conveyor belt 42 at intervals corresponding to the usage conditions, which include one or more of the usage time of the conveyor belt 42, the weight of the substrate S, and the surface roughness of the substrate S. In this control device 20, the state of the conveyor belt 42 is determined at intervals corresponding to the usage conditions, so the state of the conveyor belt 42 can be determined more appropriately.

[0039] Furthermore, the mounting apparatus 13, as a mounting-related device, comprises a transport device 40 for transporting the substrate S, a mounting unit 30 as a work unit for performing work on the substrate S, and the control device 20 described above. Similar to the control device 20 described above, this mounting apparatus 13 can address any malfunctions that may occur in the transport device 40 as early as possible.

[0040] Note that the object work system and determination method disclosed in this specification are not limited to the above-described embodiments in any way, and it goes without saying that they can be implemented in various forms as long as they belong to the technical scope of the present invention.

[0041] For example, in the above-described embodiment, the control unit 21 is configured to be in any one of the states having the damaged portion B, the soiled portion U, and the exposed portion E as the mode of the defective state of the conveyor belt 42. However, it is not particularly limited thereto, and any one or more of these may be omitted, or it may include other defective states. Further, the control unit 21 is configured to determine one or more defective states from at least two of the repair state, the soiled state, and the replacement state as the state of the conveyor belt 42. However, it is not particularly limited thereto, and any one or more of the repair state, the soiled state, and the replacement state may be omitted, or it may include other defective states, or it may not be configured to "determine one or more defects from two".

[0042] In the above-described embodiment, the state estimation model 26 is constructed based on the captured image 25 obtained by imaging the conveyor belts 42 of the plurality of transport devices 40 by the control unit 21. However, it is not particularly limited thereto, and the state estimation model 26 may be constructed based on the captured image 25 obtained by imaging only the conveyor belt 42 included in the transport device 40. Note that it is preferable to have a larger number of samples of the teacher data in order to perform more accurate determination.

[0043] In the above-described embodiment, the conveyor belt 42 is imaged by the object imaging unit 34 that images the substrate S. However, it is not particularly limited thereto, and a dedicated imaging unit for imaging the conveyor belt 42 may be provided. Note that it is preferable to image both the substrate S and the conveyor belt 42 by the object imaging unit 34 in order to simplify the configuration.

[0044] In the above-described embodiment, the control unit 21 periodically images the conveyor belt 42 to obtain the captured image 25. However, it is not particularly limited thereto, and the captured image 25 may be captured each time the substrate S is transported. Similarly, the control unit 21 may update the state estimation model 26 each time the substrate S is transported.

[0045] In the above-described embodiment, the mounting device 13 was explained as the main example of a mounting-related device, but the invention is not limited to this, and a printing device 11, a printing inspection device 12, a mounting inspection device 14, etc., may also be configured so that each device determines the malfunction state of the conveyor belt 42 of the transport device it possesses. That is, the mounting system 10 as an object handling system may include a plurality of transport devices 40, and the control unit of each device may acquire an image for each transport device 40, determine the state of each transport device 40, and notify the determination result for each transport device 40. Furthermore, in the above-described embodiment, the malfunction state of the conveyor belt 42 was determined by the control device 20 of the mounting-related device, but the invention is not limited to this, and for example, a management device 18 may determine the malfunction state of the conveyor belt 42 of each transport device 40 of the mounting system 10, or a central control device 60 may determine the malfunction state of the conveyor belt 42 of each transport device 40, or a control device other than these may determine the malfunction state of the conveyor belt 42 of the transport device 40. Furthermore, although the control device 20 has been described as updating the state estimation model 26, it is not limited to this, and the updating of the state estimation model 26 may be performed by other devices such as the control unit 60 or the management device 18.

[0046] In the above-described embodiment, the state of the conveyor belt 42 is determined by the image 25 captured by the object imaging unit 34. However, the system is not limited to this, and the state of the conveyor belt 42 may also be determined by the detection result of a detection unit that detects the conveyor belt 42 while the conveyor belt 42 is in operation. The detection unit may be a sensor capable of detecting the shape of the conveyor belt 42, such as a sensor that measures the height of the conveying surface of the conveyor belt 42.

[0047] In the above-described embodiment, the overall control device 60 constructs a state estimation model 66 by integrating the captured images 65 acquired from a plurality of object work systems such as the mounting systems 10, 10B, and 10C and their state information, and in the mounting device 13 as a specific mounting-related device, the state of the conveyor belt 42 is determined using this state estimation model 26. However, it is not particularly limited to this. For example, the mounting system 10 may include a plurality of transport devices 40, acquire a captured image 25 for each transport device 40, determine the state of the conveyor belt 42 for each transport device 40, and notify the determination result for each transport device 40.

[0048] In the above-described embodiment, the control unit 21 determines the state of the conveyor belt 42 using the state estimation model 26 constructed by machine learning. However, it is not particularly limited to this, and the state of the conveyor belt 42 may be determined using the captured image 25. At this time, the control unit 21 may use a plurality of captured images 25 obtained over time and determine the state of the conveyor belt 42 based on the difference. Also, in the above-described embodiment, the control unit 21 outputs and notifies information on the defective state of the conveyor belt 42 to the operation panel 36, the management device 18, and the overall control device 60 as a notification unit. However, it is not particularly limited to this, and this notification process may be omitted. In this control device 20 as well, if the information on the determined defect is stored, the information can be utilized.

[0049] In the above-described embodiment, the present disclosure has been described with the mounting system 10, the mounting device 13, and the control device 20. However, it is not particularly limited to this, and it may also be a control method or a determination method for mounting-related devices, and a program for realizing these methods.

[0050] Herein, the determination method of the present disclosure may be configured as follows. For example, the determination method of the present disclosure is a determination method performed by an object work system having an object-related device that performs an object-related process for mounting components onto an object that has been transported by a transport device equipped with a transport member for transporting an object on which the object is placed, and includes an acquisition step of acquiring an image of the transport member captured by an imaging unit from an imaging unit, a determination step of determining the state of the transport member based on the acquired image, and a notification step of notifying the result of the determination, wherein the determination step determines from the image of the transport member at least two to one or more defective states from among a state in which the transport member needs to be repaired, a dirty state in which dirt has adhered to the transport member, and a replacement state in which the transport member needs to be replaced.

[0051] This determination method, like the object handling system described above, can address potential malfunctions in the transport device as early as possible. Furthermore, this determination method may employ various forms of the object handling system or mounting-related equipment described above, or additional steps may be added to implement the functions of the object handling system or mounting-related equipment described above.

[0052] This specification describes the technical concept in which, in the original claim 3, "the object handling system described in claim 1" was changed to "the object handling system described in claim 1 or 2", in the original claim 6, "the object handling system described in claim 1 or 2" was changed to "the object handling system described in any one of claims 1 to 5", in the original claim 7, "the object handling system described in claim 1 or 2" was changed to "the object handling system described in any one of claims 1 to 6", and in the original claim 8, "the object handling system described in claim 1 or 2" was changed to "claims 1 to 7 Technical concepts have also been disclosed, including a change to "object work system described in any one of the following claims," ​​a change in claim 9 of the original application from "object work system described in claim 1 or 2" to "object work system described in any one of claims 1 to 8," a change in claim 10 of the original application from "object work system described in claim 1 or 2" to "object work system described in any one of claims 1 to 9," and a change in claim 11 of the original application from "object work system described in claim 1 or 2" to "object work system described in any one of claims 1 to 10."

[0053] This disclosure is applicable to the technical field of equipment that performs processes such as sampling and placement of parts.

[0054] 10, 10B, 10C Mounting system, 11 Printing device, 12 Printing inspection device, 13 Mounting device, 14 Mounting inspection device, 18 Management device, 19 Network, 20 Control device, 21 Control unit, 22 Storage unit, 23 Mounting condition information, 24 Image recording information, 25, 25a-25d Captured images, 26 State estimation model, 27 Parts supply unit, 28 Feeder, 29 Mounting imaging unit, 30 Mounting unit, 31 Head movement unit, 32 Mounting head, 33 Sampling material, 34 Object imaging unit, 35 Lifting unit, 36 Operation panel, 40 Conveying device, 41 Side frame, 42 Conveyor belt, 43 Drive unit, 44, 45 Detection unit, 47 Support base, 48 Lifting unit, 49 Support plate, 60 Overall device, 61 Control unit, 62 Storage unit, 63 64 Implementation condition information, 65 Image recording information, 66 Captured image, 67 State estimation model, 68 Display device, 69 Input device, 69 Communication unit, 70 Information display screen, 71 Information display field, 72 Message display field, B Damaged part, D Transport direction, E Exposed part, P Component, S Substrate, U Dirty part.

Claims

1. An object work system having an object-related device that performs mounting-related processing for mounting components onto an object transported by a transport device equipped with a transport member for placing and transporting an object, comprising: an acquisition unit that acquires an image of the transport member captured by an imaging unit from an imaging unit; a determination unit that determines the state of the transport member based on the acquired image; and a notification unit that notifies the result determined by the determination unit, wherein the determination unit determines from the image of the transport member at least two to one or more defect states from among a repair state in which repair is necessary for the transport member, a soiled state in which dirt is attached to the transport member, and a replacement state in which the transport member needs to be replaced.

2. The object handling system according to claim 1 comprises a plurality of transport devices, wherein the acquisition unit acquires the captured image for each transport device, the determination unit determines the state of the transport member for each transport device, and the notification unit notifies the result determined by the determination unit for each transport device.

3. The object handling system according to claim 1, wherein the determination unit uses a state estimation model constructed by machine learning based on the information of the captured image of the transport member, acquires the captured image of the transport member from the imaging unit, and applies the acquired captured image to the state estimation model to determine the state of the transport member.

4. The object work system according to claim 3, wherein the determination unit acquires an image of the transport member of the transport device taken in the implementation system and information on the defect state of the transport member determined by the operator as training data, and updates the state estimation model.

5. The object handling system according to claim 3 or 4, wherein the state estimation model is constructed based on captured images of conveying members of a plurality of conveying devices.

6. The object handling system according to claim 1 or 2, wherein the determination unit acquires an image of the transport member captured by the imaging unit that images the object being transported by the transport device.

7. The object handling system according to claim 1 or 2, wherein the determination unit periodically acquires the captured image from the imaging unit and periodically determines the state of the transport member using the state estimation model.

8. The object handling system according to claim 1 or 2, wherein the determination unit determines the state of the transport member using an image captured while the transport member is driven, or determines the state of the transport member based on the detection result by a detection unit that detects the transport member while the transport member is driven.

9. The object work system according to claim 1 or 2, wherein the acquisition unit acquires the captured images, which are video and / or still images, captured by the imaging unit from the imaging unit.

10. The object handling system according to claim 1 or 2, wherein the acquisition unit acquires the captured image of the transport member in a state where the object is not present from the imaging unit.

11. The object handling system according to claim 1 or 2, wherein the determination unit determines the condition of the transport member at intervals corresponding to the usage conditions, which include one or more of the usage time of the transport member, the weight of the object, and the surface roughness of the object.

12. A determination method to be performed by an object work system having an object-related device that performs an object-related process for mounting components onto an object transported by a transport device equipped with a transport member for transporting an object on which the object is placed, comprising: an acquisition step of acquiring an image of the transport member captured by an imaging unit from an imaging unit; a determination step of determining the state of the transport member based on the acquired image; and a notification step of notifying the result of the determination, wherein the determination step determines from the image of the transport member at least two to one or more defective states from among a state in which the transport member needs to be repaired, a dirty state in which dirt is attached to the transport member, and a replacement state in which the transport member needs to be replaced.

Images