Print inspection system

Abstract

There is described a computer implemented method comprising printing, by an industrial printer, a first mark on a first object, the first mark associated with first print data, the first print data comprising coordinates associated with the first mark to be printed in a first coordinate system; capturing, by a vision system, a first image of the printed first mark to generate first image data, the first image data comprising coordinates associated with the first printed mark in a second coordinate system; and generating, based on the first print data and the first image data, mapping data, the mapping data indicating a mapping between the first coordinate system and the second coordinate system.

Description

[0001] M&C PM364762GB

[0002] 1

[0003] Print inspection system

[0004] Technical Field

[0005] The present disclosure relates to a print inspection system.

[0006] Background

[0007] Industrial printers are robust machines designed for high-volume printing and are widely used in sectors such as manufacturing, packaging and distribution. Examples of industrial printers include continuous inkjet printers, laser printers, drop-on-demand printers, and thermal transfer printers. These printers are built to handle continuous operation and large print runs with high precision and speed. Typically, industrial printers are used in connection with a conveyor, where objects to be marked are moved along the conveyor and are marked when adjacent to the industrial printer.

[0008] Industrial printers can be used with vision systems. Vision systems typically use a camera to capture images of marks applied to objects by the industrial printer. The images can be downloaded from the vision system and inspected by the user. Following inspection, the user can make a determination as to whether the quality of the applied mark is satisfactory for their needs. However, the setting up and use of such vision systems with industrial printers is relatively complex. As such, vision systems may be incorrectly set up and / or underutilised by the user.

[0009] There provides a need for an improved print inspection system.

[0010] Summary

[0011] In a first aspect there is a computer implemented method comprising printing, by an industrial printer, a first mark on a first object, the first mark associated with first print data, the first print data comprising coordinates associated with the first mark to be printed in a first coordinate system, capturing, by a vision system, a first image of the printed first mark to generate first image data, the first image data comprising coordinates associated with the first printed mark in a second coordinate system and generating, based on the first print data and the first image data, mapping data, the

[0012] 69532054-3 M&C PM364762GB

[0013] 2 mapping data indicating a mapping between the first coordinate system and the second coordinate system.

[0014] Advantageously, generating mapping data between the first print data and first image data enables more efficient processing when comparing future prints of marks with associated print data. For example, mapping data enables print data to be modified into the same coordinate system as the image data. A comparison between images captured by the vision system and a synthetic image based on modified print data can then take place within the same coordinate system. Effects such as translation, rotation and / or scaling between the captured images and synthetic images are therefore mitigated during the comparison.

[0015] The first coordinate system may be a print data coordinate system. The second coordinate system may be an image coordinate system. The first and second coordinate systems are different coordinate systems.

[0016] The method may further comprise generating, based on the first print data and mapping data, reference data associated with the first mark to be printed. Optionally, the method may further comprise training a model using the reference data associated with the first mark to be printed, the trained model configured to process image data generated by the vision system and output quality data associated with a mark contained within the image data.

[0017] The reference data associated with the first mark to be printed may comprise a synthetic image of the first mark to be printed in the second coordinate system.

[0018] Generating the reference data associated with the first mark to be printed may comprise generating, using the first print data associated with the first mark to be printed, a synthetic image of the first mark to be printed in the first coordinate system, and projecting the synthetic image of the first mark to be printed in the first coordinate system into the second coordinate system using the mapping data to obtain the synthetic image of the first mark to be printed in the second coordinate system.

[0019] Generating the reference data associated with the first mark to be printed may further comprise modifying the synthetic image of the first mark to be printed in the second

[0020] 69532054-3 M&C PM364762GB

[0021] 3 coordinate system using scale data. For example, the scale data may be used to modify the size of the mark, characters, or individual elements, e.g. dots, that make up the mark in the synthetic image in the image coordinate system.

[0022] The method may further comprise printing, by the industrial printer, a second mark on a second object, the second mark associated with second print data, the second print data comprising coordinates associated with the second mark to be printed in the first coordinate system, generating, based on the second print data and mapping data, reference data associated with the second mark to be printed, the reference data associated with the second mark to be printed comprising coordinates associated with the second mark to be printed in the second coordinate system.

[0023] The method may further comprise capturing, by the vision system, a second image of the second printed mark to generate second image data, the second image data comprising coordinates associated with the second printed mark in the second coordinate system and comparing the reference data associated with the second mark to be printed and the second image data to determine quality data associated with the print of the second mark.

[0024] The second print data may be the same as the first print data. For example, the mark may be the same across different objects. In such cases, the reference data associated with the second mark to be printed may be the same as the reference data associated with the first mark to be printed.

[0025] Comparing the reference data associated with the second mark to be printed and the second image data to determine quality data associated with the print of the second mark may comprise using a model. The comparing may be based on comparing the shapes of the mark in the reference data, or portions of the marks, such as characters, against the shapes of the mark in the second image data.

[0026] The reference data associated with the second mark to be printed may comprise a synthetic image of the second mark to be printed in the second coordinate system.

[0027] Generating the reference data associated with the second mark to be printed may comprise generating, using the print data associated with the second mark to be

[0028] 69532054-3 M&C PM364762GB

[0029] 4 printed, a synthetic image of the second mark to be printed in the first coordinate system and projecting the synthetic image of the second mark to be printed in the first coordinate system into the second coordinate system using the mapping data to obtain the synthetic image of the second mark to be printed in the second coordinate system.

[0030] Generating the reference data associated with the second mark to be printed may further comprise modifying the synthetic image of the second mark to be printed in the second coordinate system using scale data. For example, the scale data may be used to modify the size of the mark, characters, or individual elements, e.g. dots, that make up the mark in the synthetic image in the image coordinate system.

[0031] Comparing the reference data associated with the second mark to be printed and the second image data to determine quality data associated with the print of the second mark may comprise processing the reference data associated with the second mark to be printed and the second image data using the model to obtain an output from the model and determining the quality data based on the output from the model.

[0032] The processing of the reference data associated with the second mark to be printed may be done during a training step of the model. For example, the model may be trained with the reference data associated with the second mark to be printed in order to inform the model of what the reference point is. The model, once trained may then be able to process the second image data to generate an output of quality. Alternatively, the model may take both the reference data associated with the second mark to be printed and the second image data during runtime, and may compare to generate an output of quality.

[0033] Processing the reference data associated with the second mark to be printed and the second image data using the model to obtain an output from the model may further comprise determining a first set of regions associated with the reference data associated with the second mark to be printed, the first set of regions corresponding to bounding boxes associated with the second mark to be printed, determining a second set of regions associated with the second image data, the second set of regions corresponding to bounding boxes associated with the printed second mark and processing the first regions and second regions using the model to obtain the output from the model.

[0034] 69532054-3 M&C PM364762GB

[0035] 5

[0036] That is, those regions that contain characters, as defined by bounding boxes, are identified. This means that the entire images do not need to be compared, but only those locations that correspond with bounding boxes. Of course, if it is determined that a bounding box does not fully contain a character in the second image data, the regions that are processed may be expanded.

[0037] The method may further comprise comparing the quality data to a quality threshold and determining, based on the comparing, whether the quality of the second mark satisfies the quality threshold.

[0038] The comparing of the quality data to the quality threshold may be carried out at the industrial printer. For example, the vision system may transmit the quality data to the industrial printer, and the industrial printer may compare the quality data to the quality threshold.

[0039] The method may further comprise displaying a virtual selector on a human machine interface associated with the industrial printer, the virtual selector configured to allow a selection, via user input to the human machine interface, of the quality threshold and determining the quality threshold based on a state of the virtual selector.

[0040] The method may further comprise receiving user input at the human machine interface, the user input manipulating the virtual selector to change the state of the virtual selector.

[0041] The receiving of user input may occur prior to printing the second mark.

[0042] The industrial printer may store the quality threshold.

[0043] The virtual selector may comprise a virtual slider configured to be slid along an axis when manipulated by the user, wherein the state of the virtual selector comprises the virtual slider’s position along the axis, and wherein the virtual slider’s position along the axis determines the quality threshold.

[0044] 69532054-3 M&C PM364762GB

[0045] 6

[0046] The method may further comprise displaying on the human machine interface a predicted image of a mark having a quality associated with the determined quality threshold.

[0047] The mark shown in the predicted image may be the second mark, or may be a generic mark, such as a list of random characters.

[0048] The human machine interface may be coupled to the industrial printer. That is, the human machine interface may be integrated into the industrial printer.

[0049] The first mark and second mark may be associated with the same print design specification.

[0050] That is, the first mark, used for calibration, is a mark according to the user’s print job. As such, it is not necessary to print a standard calibration pattern that will need to be discarded following calibration. That is, the first mark may not be a known calibration pattern.

[0051] The first and second coordinate systems may be two dimensional coordinate systems.

[0052] The mapping data may comprise a transformation matrix.

[0053] In a second aspect there is a system comprising an industrial printer and a vision system. The industrial printer is configured to print a first mark on a first object, the first mark associated with first print data, the first print data comprising coordinates associated with the first mark to be printed in a first coordinate system, transmit the first print data to the vision system, print a second mark on a second object, the second mark associated with second print data, the second print data comprising coordinates associated with the second mark to be printed in the first coordinate system, and transmit the second print data to the vision system. The vision system is configured to capture a first image of the printed first mark to generate first image data, the first image data comprising coordinates associated with the first printed mark in a second coordinate system, generate, based on the first print data and the first image data, mapping data, the mapping data indicating a mapping between the first coordinate system and the second coordinate system, generate, based on the second print data

[0054] 69532054-3 M&C PM364762GB

[0055] 7 and mapping data, reference data associated with the second mark to be printed, the reference data associated with the second mark to be printed comprising coordinates associated with the second mark to be printed in the second coordinate system, capture a second image of the second printed mark to generate second image data, the second image data comprising coordinates associated with the second printed mark in the second coordinate system and compare, using a model, the reference data associated with the second mark to be printed and the second image data to determine quality data associated with the print of the second mark.

[0056] In a third aspect there is a computer program comprising instructions, which when the program is executed by an industrial printer, cause the industrial printer to print a first mark on a first object, the first mark associated with first print data, the first print data comprising coordinates associated with the first mark to be printed in a first coordinate system, transmit the first print data to a vision system, print a second mark on a second object, the second mark associated with second print data, the second print data comprising coordinates associated with the second mark to be printed in the first coordinate system, transmit the second print data to the vision system. When the program is executed by a vision system, the program causes the vision system to the vision system configured to capture a first image of the printed first mark to generate first image data, the first image data comprising coordinates associated with the first printed mark in a second coordinate system, generate, based on the first print data and the first image data, mapping data, the mapping data indicating a mapping between the first coordinate system and the second coordinate system, generate, based on the second print data and mapping data, reference data associated with the second mark to be printed, the reference data associated with the second mark to be printed comprising coordinates associated with the second mark to be printed in the second coordinate system, capture a second image of the second printed mark to generate second image data, the second image data comprising coordinates associated with the second printed mark in the second coordinate system, and compare, using a model, the reference data associated with the second mark to be printed and the second image data to determine quality data associated with the print of the second mark.

[0057] In a fourth aspect there is a computer implemented method comprising displaying a virtual selector on a human machine interface associated with an industrial printer, the virtual selector configured to allow a selection, via user input to the human machine

[0058] 69532054-3 M&C PM364762GB

[0059] 8 interface, of a quality threshold associated with a print of a mark by the industrial printer, and determining a quality threshold associated with the print of the mark by the industrial printer based on a state of the virtual selector.

[0060] The method may further comprise receiving user input at the human machine interface, the user input manipulating the virtual selector to change the state of the virtual selector.

[0061] The virtual selector may further comprise a virtual slider configured to be slid along an axis when manipulated by the user, wherein the state of the virtual selector comprises the virtual slider’s position along the axis, and wherein the virtual slider’s position along the axis determines the quality threshold.

[0062] The method may further comprise displaying on the human machine interface a predicted image of the mark having a quality associated with the determined quality threshold.

[0063] When displaying the virtual selector on the human machine interface, and prior to receiving user input, the quality threshold may comprise a default value.

[0064] The human machine interface may comprise a touch screen.

[0065] The quality threshold may be used to compare with a determined quality of a mark. The determined quality of the mark may be obtained by processing image data, the image data comprising a captured image of the printed mark. The image data may be generated by a vision system. The processing of the image data to determine a quality of the mark may be carried out by the vision system. The determined quality of the mark may be sent to the industrial printer. The industrial printer may compare the determined quality of the mark to the quality threshold. The industrial printer may assign a pass or a fail to the mark based on whether the determined quality satisfies the quality threshold. The industrial printer may output on the human machine interface whether the mark passed or failed.

[0066] The fourth aspect may be combined with any of the first to third aspects.

[0067] 69532054-3 M&C PM364762GB

[0068] 9

[0069] In a fifth aspect, there is a data processing apparatus comprising means for carrying out the steps of any of the methods of the first to fourth aspects.

[0070] In a sixth aspect, there is a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the steps of any of the methods of the first to fourth aspects.

[0071] In a seventh aspect, there is a computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to carry out the steps of any of the methods of the first to fourth aspects.

[0072] Brief Description of Figures

[0073] The present disclosure will now be further described by way of example only with reference to the accompanying drawings, in which:

[0074] Figure 1 is a schematic diagram of a printing system according to the present disclosure;

[0075] Figure 2 is a flow diagram of a method according to the present disclosure;

[0076] Figure 3 is a schematic diagram of print data used in the printing system of Figure 1 ;

[0077] Figure 4 is a flow diagram of a method according to the present disclosure; and

[0078] Figure 5 is an example display on a human machine interface according to the present disclosure.

[0079] Detailed

[0080] With reference to Figure 1 , there is shown a schematic diagram of a printing system 100 comprising a vision system 102 and an industrial printer 104.

[0081] The vision system 102 comprises an image capture device 106, a processor 108, a memory 110, and a vision system I / O interface 112. The image capture device 106 is

[0082] 69532054-3 M&C PM364762GB

[0083] 10 configured to capture digital images of a scene within the image capture device’s 106 field of view FOV. While not shown, the image capture device 106 comprises those features necessary to capture digital images, such as a lens, shutter, aperture, and image sensor. For example, the image capture device 106 may be a digital camera. The memory 110 may be in the form of a non-volatile memory, such as a solid state drive or hard disk drive. The memory 110 may be in the form of volatile memory, such as random access memory (RAM). The memory 110 may comprise both non-volatile memory and volatile memory. While not shown, the memory 110 comprises computer readable instructions that, when executed by the processor 108, cause the processor 108 to control the vision system 102 according to the methods described herein.

[0084] The industrial printer 104 comprises marking hardware 114, a processor 116, memory 118, a printer I / O interface 120, and a human machine interface (HMI) 122. The marking hardware 114 comprises any suitable hardware for printing (also referred to as applying) a mark to an object 150a, 150b. For example, the marking hardware 114 may comprise a print head for directing ink 155 from an ink reservoir (not shown) onto objects 150a, 150b so as to apply a mark to the objects 150a, 150b. Alternatively, the marking hardware 114 may comprise a laser source and one or more mirrors and lenses for directing the laser light onto the objects 150a, 150b in order to apply a mark. While any industrial printer may be used, the following example implementations will be described in the context of a continuous inkjet printer (CM), where the mark to be applied comprises a number of individual dots, each dot corresponding to a droplet of ink. The memory 118 may be in the form of a non-volatile memory, such as a solid state drive or hard disk drive. The memory 118 may be in the form of volatile memory, such as random access memory. The memory 118 may comprise both non-volatile memory and volatile memory. While not shown, the memory 118 stores computer readable instructions that, when executed by the processor 116, cause the processor 116 to control the industrial printer 104 according to the methods described herein. While not shown, the industrial printer 104 may be configured to remotely connect to a remote server, e.g. a cloud server. The industrial printer 104 may upload data to the remote server, such as image data, quality data, print data, and / or a print design specification. Uploading data to the remote server allows the data to be remotely stored and remotely accessed by a user. The HMI 122 may be in any suitable form that enables the user to interact with the industrial printer 104 and vision system 102. For example, the HMI 122 may be in the form of a touch screen, which can both output a

[0085] 69532054-3 M&C PM364762GB

[0086] 11 visual display for the user and receive input via the user’s touch. Alternatively, the HMI 122 may comprise a computer display for visually outputting data, and may have a separate input mechanism, such as a keyboard, for inputting commands or data to the HMI 122.

[0087] It will be appreciated that the vision system 102 and industrial printer 104 are depicted in a simplified way to aid understanding of the present disclosure, and that each of the vision system 102 and industrial printer 104 may include more components over that which is shown.

[0088] The industrial printer 104 is configured to apply a mark to an object 150b according to a print design specification 130. The print design specification 130 is associated with a print job that the user wishes to execute, and comprises data that specifies the parameters of the mark to be printed. For example, a print job may be to print a sell by date on objects 150a, 150b. The print design specification 130 comprises all information necessary for the industrial printer 104 to apply the desired mark. For example, the print design specification 130 comprises coordinate data, the coordinate data comprising coordinates associated with the mark to be printed. The coordinate data may comprise the locations of the individual elements of the mark, such as the individual dots that make up the mark and / or the coordinates of bounding boxes of individual characters that make up the mark. The coordinate data exists in a first coordinate system, referred to herein as a print data coordinate system. The print data coordinate system is a two dimensional coordinate system. The industrial printer 104 is able to use the coordinate data to apply the elements of the mark, such as the individual dots that make up the mark, in the correct location relative to the other elements of the mark.

[0089] The vision system 102 is coupled to the industrial printer 104 via connection 124 between the vision system I / O interface 112 and the printer I / O interface 120. Connection 124 may be a wired connection and / or wireless connection. Connection 124 enables data to be transferred between the vision system 102 and the industrial printer 104. Any suitable protocol may be used to transfer data between the vision system 102 and the industrial printer 104. In a specific implementation, the protocol is the WebSocket protocol, standardized in RFC 6455. The vision system 102 may also be powered via connection 124. That is, the vision system 102 may obtain electrical

[0090] 69532054-3 M&C PM364762GB

[0091] 12 power via the industrial printer 104. Alternatively, the vision system 102 may obtain electrical power independently from the industrial printer 104. The printer I / O interface 120 may be in the form of a junction box, which is mounted on an external surface of the industrial printer 104. The junction box may include one or more sockets for coupling the vision system 102 to the industrial printer 104. While a single connection 124 is shown, it will be appreciated that connection 124 may comprise multiple connections. For example, connection 124 may comprise a first physical wire that transmits power to the vision system 102 and a second physical wire that transmits data between the vision system 102 and the industrial printer 104. The second physical wire may be replaced with a wireless connection. Alternatively, a single physical wire may be used to both transmit power to the vision system 102 and allow data to be transferred between the vision system 102 and the industrial printer 104. The junction box may be directly integrated into the marking hardware 114, such as a print head, of the industrial printer 104.

[0092] As will be described in more detail below, the industrial printer 104 is configured to transmit print data 140 to the vision system 102, and the vision system is configured to transmit quality data 160 to the industrial printer 104. Briefly, print data 140 provides information to the vision system 102 about what is to be printed according to the print design specification 130. The vision system 102 is configured to use the print data 140 to determine a quality associated with a print of a mark captured in an image by the visions system. The visions system generates the quality data 160 which provides information to the industrial printer 104 about the quality of the print of the mark applied by the industrial printer 104. The quality data 160 can be compared with a threshold value and / or output to the user on the HMI 122.

[0093] The printing system 100 is configured to be used in an industrial setting. Typical examples of industrial settings are warehouses, factories, depots, etc. Figure 1 schematically illustrates an example of an industrial setting in which objects 150a, 150b to be marked are advanced in direction A past the industrial printer 104 using a conveyor 152. Only two objects 150a, 150b are shown in Figure 1 for clarity, but it will be appreciated that there may be any number of objects to be marked arranged along the conveyor 152. When a given object is adjacent the industrial printer 104, such as object 150b in Figure 1 , the industrial printer 104 prints the mark on the object 150b. In such an arrangement, the vision system 102 is mounted with respect to the conveyor

[0094] 69532054-3 M&C PM364762GB

[0095] 13

[0096] 150 downstream of the industrial printer 104, or at least such that its FOV covers a scene in which a marked object will appear. Any suitable mounting may be used. For example, the vision system 102 may be mounted directly to the industrial printer 102. The vision system 102 may be mounted to the marking hardware 114, such as mounted to the print head. Alternatively, the vision system 102 may be mounted to a bracket (not shown) coupled to the conveyor 152, or coupled to another object that remains stationary with respect to the conveyor 152.

[0097] As described above, the vision system 102 is configured to generate image data by using the image capture device 106 to capture images of marks that have been applied to objects 150a, 105b by the industrial printer 104. In the implementation shown in Figure 1, a first object 150a has been marked and is now adjacent the image capture device 106. When the first object 150a to which the mark has been applied is adjacent the image capture device 106, the vision system 102 captures an image of the mark. A suitable trigger mechanism may be used to control when the vision system 102 captures the image. For example, a trigger may be sent from the industrial printer 104 to the vision system via connection 124, the trigger causing the vision system to capture the image. The industrial printer may generate the trigger, or the trigger may be generated by an external sensor. For example, the industrial printer 104 may have a sensor, such as a photocell, that is used to detect the object 150a, 150b. Upon detection of the object 150a, 150b, the industrial printer 104 applies the mark to the object 150a, 150b, and then sends a trigger to the vision system 102. Alternatively, the vision system 102 may comprise its own sensor, such as a photocell, which is used to detect the object 150a, 150b independently of the industrial printer 104. Advantageously, providing the vision system with an independent sensor to detect the object 150a, 150b, allows the vision system 102 to capture images of any unmarked objects that have been missed by the industrial printer 104. That is, if the sensor used by the industrial printer 104 does not trigger when an object 150a, 150b is adjacent to the industrial printer 104, the object will not be marked. The vision system 102, having an independent method of detecting adjacent objects 150a, 150b, allows the capture and identification of missed prints. While not shown, one or more illumination devices, such as lights, may be mounted on the industrial printer 104, and / or vision system 102, which can be used to provide illumination for the vision system 102 when capturing images. Any suitable arrangement may be used such that the vision system 102 is able to capture images of the applied mark.

[0098] 69532054-3 M&C PM364762GB

[0099] 14

[0100] The image data generated by the vision system 102 may be in the form of a digital image file. That is, the image data may comprise pixel data, the pixel data specifying a value for each pixel of the image. The image data may be in any suitable format, such as a RAW format, JPEG format, TIFF format, PNG format, GIF format, and BMP format. For example, the image data may be initially generated in the RAW format, and may be converted to a compressed format, such as a JPEG format, for transfer or storage. The image data generated by the vision system 102 has an associated coordinate system, referred to herein as an image coordinate system. Marks contained within the image will have an associated location within the image coordinate system. The image coordinate system is a two dimensional coordinate system.

[0101] The image coordinate system differs from the print data coordinate system, and so to compare marks in the print data coordinate system with marks in the image coordinate system, it is advantageous to perform a calibration routine. The calibration routine can be carried out automatically upon the initialization of a new print job (e.g. when a new print design specification 130 is loaded into the industrial printer 104 or executed by the industrial printer 104), or can be manually instructed by the user using the HMI 122. Once initiated, the calibration is fully automatic, requiring no further input from the user. The calibration routine will now be described, with reference to Figure 2.

[0102] In a pre-calibration step, the vision system 102 may carry out an automatic initial configuration routine, where focus, exposure, and gain values for the image capture device 106 are automatically determined by the vision system 102. Such automatic configuration routines for digital cameras are notoriously well known and are not described here.

[0103] At step 200, the industrial printer 104 prints a first mark on a first object, such as object 150a in Figure 1. The first mark is to be used for calibration, but may be a mark in accordance with the print design specification 130 associated with the print job that the user wishes to execute. Advantageously, by printing a mark on an object according to the user’s print job, it is not necessary to print a known calibration pattern that will end up being discarded by the user. Additionally, the vision system 102 can be easily recalibrated mid production (e.g. without stopping the production line to add a calibration target). Of course, it is still possible that in an implementation the first mark

[0104] 69532054-3 M&C PM364762GB

[0105] 15 could be a known calibration pattern. As described above, the print design specification 130 specifies coordinates associated with the first mark to be printed in a print data coordinate system. That is, each element (e.g. dot) of the first mark will have coordinates that are defined in the print data coordinate system.

[0106] At step 202, a first image of the printed first mark is captured by the image capture device 106 of the vision system 102 to generate first image data 170. As described above, objects within images captured by the image capture device 106 will be associated with the image coordinate system. As such, the first image data 170 depicts a scene of the first mark in the image coordinate system. That is, each element (e.g. dot) of the first mark in the first image data 170 will have a coordinates that are defined in the image coordinate system.

[0107] At step 204, the industrial printer 104 transmits the print data 140 to the vision system 102. As described above, print data 140 is data relating to the print design specification 130. The print data 140 specifies the coordinates associated with the mark to be printed in the print data coordinate system. The print data 140 may be the print design specification 130, or may be a portion of the print design specification 130, or may be data derived from the print design specification 130. In a particular implementation, the vision system 102 is able to use the print data 140 to generate a synthetic image of the mark, or a portion of the mark, that is to be printed in the print data coordinate system, as will be described later.

[0108] Figure 3 shows a schematic illustration of a specific implementation of the print data 140. In this example, the print data 140 comprises shape data, such as font data 302, bounding box data 304, and string data 306.

[0109] Font data 302 comprises information relating to the font used according to the print design specification 130, e.g. what font the characters of the mark are printed in. The font data specifies the shape of each character in the set of characters (such as a-z, A- Z, 0-9, etc.) according to the particular font used. The shape of each character may be specified in the form of the relative location of each dot that makes up each character according to the font. Rather than the font data 302 specifying the specific shape of each character in the set of characters, the font data 302 may indicate a font that is being used and the vision system 102 may use the indication of the font that is being

[0110] 69532054-3 M&C PM364762GB

[0111] 16 used to obtain data specifying the shape of each character from its memory 110. That is, the memory 110 may already comprises data relating to a number of different fonts, and the font data 302 sent to the vision system 102 may be an indication of which font is being used to mark the object 150a, 150b.

[0112] Bounding box data 304 comprises the coordinates, in the print data coordinate system, of each bounding box associated with each character to be printed according to the print design specification. A bounding box is an area in which an individual character is located. Any suitable coordinate associated with each bounding box may be used. For example, the coordinates of each top left corner of each bounding box may be included in the bounding box data 304. Alternatively, the coordinates of all corners of each bounding box may be included in the bounding box data 304. The bounding box data 304 may also comprise a size of each bounding box in the print data coordinate system. The bounding box data 304 may also comprise the shape of each bounding box. For example, the bounding box data 304 may, for each bounding box, specify a single coordinate of the bounding box in the print data coordinate system, the shape of the bounding box, and / or the width and height of the bounding box. Of course, if all four coordinates of all four corners of each bounding box are included in the bounding box data 304, it is not necessary to include the shape or size of the bounding box, since this can be derived from the four coordinates.

[0113] String data 306 comprises a string of the specific characters to be printed according to the print design specification 130. The string data 306 may be in any suitable format, such as a HEX format.

[0114] Using the font data 302, bounding box data 304 and the string data 306, the vision system 102 is able to generate a synthetic image of the mark, or a portion of the mark, in the print data coordinates system. For example, the vision system 102 may use the bounding box data 304 to arrange the bounding boxes in their relative locations in the print data coordinate system, determine the specific character for each bounding box using the string data 306, and determine the shape of each character that is to be inserted in each bounding box using the font data 302. The determination as to which specific character is to be placed in which bounding box may be based on determining the ordering the bounding boxes and characters. For example, the first character in the string will appear in the first bounding box, the second character in the second

[0115] 69532054-3 M&C PM364762GB

[0116] 17 bounding box, etc. Alternatively, the string data 306 and the bounding box data 304 may comprise identification data that can be used to match individual characters with individual bounding boxes.

[0117] In other implementations, such as where the print job relates solely to a logo, the print data 140 may instead comprise different information to that described above. For example, string data 306 may not be necessary if the logo is a collection of dots that make up a non-character graphical image. In such cases, a single bounding box relating to the position of the logo may be provided, along with shape data of the logo, specifying the shape of the logo.

[0118] While a specific form of print data 140 has been described, it will be appreciated that the print data 140 may take any suitable form, such that the vision system 102 is able to generate synthetic image of the mark. The print data 140 may comprise other data, such as orientation data, inverse data, mirror data and / or bold data. Orientation data may relate to the relative orientation of the vision system 102 and the printed mark. For example, the industrial printer 104 may apply a mark, or a portion of the mark, upside down with respect to the vision system 102. Inverse data may specify that an inverse mark has been applied. For example, when inverse printing, the individual characters that make up the mark themselves do not consist of dots printed by the industrial printer 104, but rather the background of the characters is printed with dots, so that the characters are created from the non-printed areas. Mirror data relates to whether the mark has been printed as a mirror image. Bold data relates to whether any of the characters are in a bold font, where a bold font typically uses double the number of horizontal dots used to make up each character when compared with a non-bold font.

[0119] It will be appreciated that ordering of steps 202 and 204 may be reversed. That is, the industrial printer 104 may transmit the print data 140 to the vision system 102 before applying the mark to the object 150a.

[0120] Turning back to Figure 2, at step 206, the vision system 102 generates mapping data 180 based on the print data 140 and the first image data 170. The mapping data 180 is used to convert from the print data coordinate system to the image coordinate system. The mapping data 180 may be in the form of a transformation matrix, such as a homogeneous transformation matrix. The mapping data 180 may apply linear

[0121] 69532054-3 M&C PM364762GB

[0122] 18 transformations (rotation, scaling and translation) to a point in one coordinate system to map it into another coordinate system. The mapping data 180 may be generated using any suitable method. For example, the vision system 102 may generate, using the print data 140, a synthetic image of the first mark that is to be printed in the print data coordinate system. The vision system 102 may then compare the synthetic image to the first image data 170. Pattern recognition techniques may be used to determine areas of the synthetic image that correspond with areas of the first image data 170. A mapping between the two coordinate systems can then be determined.

[0123] When generating the mapping data 180, the vision system 102 may determine scale data 190. The scale data 190 is data relating to the size of components that make up the mark, or of the mark itself, in the image coordinate system. For example, in the case of a CIJ printer, each printed character is made up of many individual dots. In such a case, the scale data 180 may comprise the diameter of the printed dots of the first mark in the first image data 170. An average value may be used. That is, while the dots imparted on the object 150a, 150b will all generally be the same size, some dots within the first image data 170 may appear larger than others due to lens distortion. In such cases, an average value of the dot diameter may be selected. The scale data 190 may also comprise the distance between individual adjacent dots. For example, the distance may be the measure of separation between the centres of two adjacent dots. An average value may be used. For example, the distance between adjacent dots may be measured for multiple pairs of dots, and an average value taken to be the distance between adjacent dots. Averaging in this way mitigates effects from lens distortion. Knowing the distance between individual adjacent dots enables the calculation of the width and height of characters in the image coordinate system.

[0124] The scale data 190 can be obtained from measuring the values directly from the first image data 170. However, in some cases, the values of the scale data 190 may be estimated. This may be the case where it is difficult to accurately measure the dot diameter and dot spacing from the first image data 170. For example, when the first mark is applied by the industrial printer 104, some of the dots may touch one another, making it difficult to measure the exact distance between them and / or determine their diameter. If the first mark is applied on a surface of the object 150a that already comprises graphics, these background graphics may make accurate measurements of the dot diameter and dot spacing difficult to measure. The scale data 190 may be

[0125] 69532054-3 M&C PM364762GB

[0126] 19 estimated by selecting initial values for the scale data 190, and comparing to the first image data 170. The estimate of the scale data 190 may be refined until an acceptable value is found.

[0127] As will be described in more detail later, the scale data 190 may be used with the mapping data 180 to generate a synthetic image of the mark to be printed in the image coordinate system. For example, positions of individual dots in a synthetic image of the mark in the print data coordinate system may be modified using the mapping data 180, such that the positions of the individual dots are projected into the image coordinate system. The size of the individual dots may then may be correctly scaled according to the scale data 190.

[0128] Optionally, at step 208, the vision system 102 generates reference data 174 based on the print data 140 and mapping data 180. The reference data 174 is data that indicates what the desired mark, or a portion of the desired mark, would look like when observed by the vision system 102. For example, the reference data 174 may be a synthetic image of the first mark in the image coordinate system, or may be individual synthetic images of each character that is used in the mark. If the print data 140 comprises information for multiple marks, then multiple synthetic images, or portions thereof, of multiple marks may be generated. That is, if the print data 140 comprises information for not just the first mark, but for one or more subsequent marks, then reference data (e.g. a synthetic image) may be generated for each mark, and / or for each character of each mark.

[0129] As noted above, the print data 140 allows the vision system 102 to create a synthetic image of the mark to be printed in the print data coordinate system. The reference data 174 may be generated by modifying the synthetic image of the mark to be printed in the print data coordinate system using the mapping data 180 to obtain a synthetic image of the mark to be printed in the image coordinate system. For example, the location of each element, e.g. each dot, of the mark specified in the print data coordinate system is projected into the image coordinate system using the mapping data 180 to obtain the location of each dot in the reference data 174. The location of the centre of each dot may be used as the coordinates for that dot. The size of each dot in the reference data 174 may then be sized accordingly using the scale data 190. This ensures that the

[0130] 69532054-3 M&C PM364762GB

[0131] 20 scale of the dots in the reference data 174 accurately reflects the scale that will be observed in an image captured by the vision system 102

[0132] Optionally, at step 210, the vision system 102 trains, or configures, a model 132 using the reference data 174. During normal runtime, the model 132 is configured to process image data generated by the vision system 102 and output quality data 160. The model 132 may be any suitable model. An example of a suitable model 132 is a HALCON® vision model produced by MVTec Software Gmbh. When using a HALCON® vision model, either an NCC or shape model may be used. The model 132 may be loaded into the memory 110 of the vision system, and the processor 108 may train, or configure, the model 132 based on the reference data 174. In this way, the model 132 is trained on synthetic images that display the mark in the image coordinate system, rather than being trained on synthetic images that display the mark in the print data coordinate system. As such, the training data is more representative of the data that the model 132 will process during runtime, e.g. is more representative of what will actually be captured by the vision system 102. When using a Halcon® vision model the training of the model 132 may lead to the generation of multiple shape models. For example, for each character, or logo, that is indicated in the print data 140, a specific shape model may be generated for that character, or logo. In this way, if the print data 140 indicates that the print design specification 130 will contain thirty different characters in a selected font, the configuration, or training, of the model 132 may comprise generating thirty shape models, one for each character. Each shape model is then configured to a specific character. Optionally, when using a Halcon® vision model the training may also be used to configure the number of pyramid levels that will be used when carrying out shape detection. In cases where the model 132 is a classifier, the reference data 174 may be used as ground truth and used in the training of the classifier.

[0133] As described above, quality data 160 is data indicative of the quality of the mark contained within image data (such as the first image data 170). The quality data 160 may be defined in any suitable form, such as a score between 0 and 1. A value of 1 may indicate a perfect quality and a value of 0 may indicate a poor quality.

[0134] While the model 132 has been described as being trained on reference data 174, in other implementations the model 132 may not require any specific training or

[0135] 69532054-3 M&C PM364762GB

[0136] 21 configuration based on the reference data 174. For example, the model 132 may be a general comparison model that can take as input the reference data 174 and image data, and output a score (such as a numerical value) indicative of the similarity between the reference data 174 and image data. The score may be used as the quality data.

[0137] Once the calibration routine is completed, the vision system 102 may send a signal to the industrial printer 104 to indicate that calibration has been completed. Additionally, the quality data 160 associated with the quality of the first printed mark contained within the first image data 170 may also be transmitted to the industrial printer 104. That is, during calibration, the model 132 may process the first image data 170 and reference data 174 associated with the first mark in order to determine the quality data 160 associated with the first image data 170. The quality data 160 may be presented to the user on the H Ml 122 of the industrial printer 104, along with an image of the printed first mark. This is described in more detail below with respect to Figure 5.

[0138] With reference to Figure 4, there is described a method of using the vision system 102 to monitor the print quality of marks printed by the industrial printer 104. The method shown in Figure 4 describes steps associated with monitoring the quality of a second mark on a second object, but it will be appreciated that the method steps may repeat for each subsequent object to be marked.

[0139] At step 400, the industrial printer 104 prints a second mark on the second object 150b. For example, the second mark may be the next mark to be applied to the next object 150b on the conveyor after the first object 150a. The second mark is in accordance with the print design specification 130 associated with the print job that the user wishes to execute.

[0140] At step 402, a second image of the printed second mark is captured by the image capture device 106 of the vision system 102 to generate second image data 172. As described above, marks within images captured by the vision system 102 will be associated with the image coordinate system. As such, the second image data 172 depicts a scene of the second mark in the image coordinate system.

[0141] 69532054-3 M&C PM364762GB

[0142] 22

[0143] At step 404, if required, the industrial printer 104 transmits print data 140 associated with the second mark to the vision system 102. The print data 140 associated with the second mark is data relating to the print design specification 130 associated with the second mark. If the second mark is identical to the first mark, such as a date that is not going to change between objects 150a, 150b, then it may not be required to transmit the print data 140 associated with the second mark, since the vision system 102 can use the previously transmitted print data 140 associated with the first mark. Furthermore, if the print data 140 sent during calibration contains information on all of the marks to be printed, despite each mark being different, further print data need not be sent. For example, if the print job relates to a mark indicating a counter, where the counter is incremented by one for each mark, the print data 140 sent during calibration may have comprised this information. That is, the print data 140 sent during calibration may have included data relating to each of the characters for each mark to be printed, as well as the data relating to the first mark.

[0144] The order of steps 402 and 404 may be reversed.

[0145] At step 406, the vision system 102 generates reference data 176 associated with the second mark. The reference data 176 is based on the print data 140 associated with the second mark and mapping data 180. As described above with respect to reference data 174 associated with the first mark, the reference data 176 associated with the second mark is data that indicates what the desired mark, or a portion of the desire mark, would look like when observed by the vision system 102. For example, the reference data 176 associated with the second mark may be a synthetic image of the second mark in the image coordinate system, or may be individual synthetic images of each character that is used in the second mark. The reference data 176 associated with the second mark may be generated by first generating a synthetic image of the second mark in the print data coordinate system using the print data 140, and then modifying the synthetic image of the second mark in the print data coordinate system using the mapping data 180 to obtain the synthetic image of the second mark in the image coordinate system. The scale data 190 may also be used to generate the reference data 176 associated with the second mark. For example, once the synthetic image of the second mark in the print data coordinate system has been modified using the mapping data 180, the individual elements, e.g. dots, of the mark may be scaled using the scaling data 190.

[0146] 69532054-3 M&C PM364762GB

[0147] 23

[0148] Step 406 may be carried out during the calibration routine (e.g. at step 208). That is, if the print data 140 sent to the vision system 102 contains information on the second mark, then the reference data 176 associated with the second mark may be generated during calibration (e.g. prior to a full production run).

[0149] At step 408, the vision system 102 uses the model 132 to process the second image data 172 and reference data 176 associated with the second mark to determine quality data 160 associated with the second printed mark. For example, the model 132 may take as input the reference data 176 associated with the second mark (e.g. a synthetic image of the second mark in the image coordinate system) and the second image data 172 (e.g. the captured image of the second printed mark), and output a score indicative of the quality of the second printed mark. In another example, the model 132 may have been trained on the reference data 176 associated with the second mark during the calibration routine, such that the model may only need to process the second image data 172 at runtime. That is, the model 132 may already be preconfigured during calibration to determine a quality of the second mark in the second image data 172.

[0150] As noted above, the quality data 160 may comprise a score. For example, the model 132 may assign a score to each character in the second mark in the second image data 172 based on the shape of each character in the second mark relative to the expected shape as defined in the reference data 174. Each score for each character may be combined to obtain an overall score for the mark.

[0151] In a specific implementation, the model 132 may only process those areas of the second image data 172 and reference data 176 associated with the second mark that contain the mark. For example, only the regions surrounded by bounding boxes may be processed. That is, the regions of the second image data 172 and the reference data 176 associated with the second mark that correspond with the bounding boxes may be extracted and those specific regions processed by the model 132. The locations of the bounding boxes are provided in the bounding box data 304 in the print data coordinate system. These locations can be translated into the image coordinate system using the mapping data 180. This then gives the locations in the second image data 172 of the regions where the individual characters are likely to be located. These regions can be extracted from the second image data 172 and processed by the model

[0152] 69532054-3 M&C PM364762GB

[0153] 24 along with the corresponding regions from the reference data 176 associated with the second mark.

[0154] In a specific implementation, the coordinates included in the bounding box data 304, which are in the print data coordinate system, are projected into the image coordinate system using the mapping data 180. If the bounding box data 304 comprises the coordinates of each corner of each bounding box (e.g. all four corners of each bounding box), then when each corner is transformed into the image coordinate system, it is possible to determine the coordinates that bound the area of the bounding box in the image coordinate system. If the bounding box data 304 comprises only one coordinate of each bounding box, such as the top left corner of each bounding box, additional information is required to determine the coordinates that bound the area of the bounding box in the image coordinate system. For example, the width of each character for each bounding box may be determined and used to define the width of the bounding box. This can be achieved by using the string data 306, font data 302 and the scale data 190. Each character for each bounding box can be determined using the string data 306. The width and height of each character can be determined using the font data 302 and scale data 190. That is, the width of each character in dots can be determined in the image coordinate system. Using the determined width and height, and using a single coordinate for each bounding box, the region bounded by each bounding box in the image coordinate system can be determined.

[0155] This then provides the specific areas in the image coordinate system at with each character of the second mark in the second image data 172, and reference data 176 associated with the second mark, should be located. As such, the vision system 102 needs only to process those specific areas in the second image data 172 that are associated with the location of the bounding boxes in the image coordinate system. This significantly reduces the processing overhead, since the entire second image data 172 does not need to be processed.

[0156] When the model 132 comprises multiple shape models, one for each character or logo, each character in the second image data 172 is processed using its corresponding shape model. For example, if in the print data 140 it is indicated that a string to be printed is “ABC“, and the bounding box data indicates the relative positions of each character A, B, and C, then a shape model for character “A” is used to process the

[0157] 69532054-3 M&C PM364762GB

[0158] 25 portion of the second image data 172 that is bounded by the bounding box at the position at which the character “A” should be in the second image data 172. Similarly, shape models for characters “B” and “C” are used to process the portions of the second image data 172 that are bounded by the bounding boxes at the position at which the characters “B” and “C” should be in the second image data 172 respectively. Each shape model is generated (or trained) based on the associated reference data. Each shape model may output a quality score, which indicates how close a given character of the printed mark matches an ideal shape, as indicated in the reference data. Each quality score for each character of the mark may be combined, and / or averaged, to obtain the quality data 160.

[0159] It may be required to modify the location of the determined bounding boxes in the image coordinate system over time. For example, due to slight deviations in the production process, the location of a mark within a captured image may drift over time when compared with the location of the mark in the image captured during calibration. The drift may be a lateral drift and / or may be an angular drift. That is, the position at which the mark was located within a captured image during the calibration routine may differ from the position at which a future mark is located within a future captured image during a production run. Slight changes mean that the mapping data 180, when used to transform bounding boxes into the image coordinate system, may not align with the locations of the characters of the printed mark in the captured image.

[0160] Such drift may be automatically detected and corrected. For example, if the calculated location of bounding boxes no longer fully contain a character in a captured image of a mark, this provides an indication that the mapping data 180 is no longer accurate. To compensate for this drift, the vision system 102 may need to apply a lateral or angular shift to the coordinates of the generated bounding boxes in the image coordinate system such that the bounding boxes align with the printed characters in the image. Any suitable method may be used. For example, when there has been a lateral drift, a character at an edge of the mark may be identified in the captured image. The character may be the first character of the mark when read from the left. The lateral difference between the position of the character in the image and its associated bounding box may be determined. The lateral difference may be used to shift each bounding box such that each bounding box becomes realigned with their respective

[0161] 69532054-3 M&C PM364762GB

[0162] 26 characters. Alternatively, or additionally, the calibration routine may be rerun to recalibrate the mapping data 180 to account for the drift.

[0163] At step 410, the vision system 102 outputs the quality data 160 associated with the second printed mark. For example, the vision system 102 transmits the quality data 160 associated with the second printed mark to the industrial printer 104. Optionally, the vision system may also transmit the second image data 172 to the industrial printer 104. The second image data 172 may have a relatively large file size, and so it may be beneficial to only send portions of the second image data 172, such as those falling within the bounding box regions. In some cases, it may be beneficial to only send a subset of the second image data 172, such as the image data generated for every 5th, 10thor 20thcaptured image. Additionally, it may be beneficial to compress the second image data 172 before sending. For example, if the second image data 172 is in the RAW format, the second image date may be converted to the JPEG format before sending.

[0164] The industrial printer 104 may take a number of different actions based on receipt of the quality data 160 and / or second image data 172. The industrial printer 104 may compare the quality data 160 to threshold quality data 185. If the quality data 160 satisfies the threshold quality data 185, the industrial printer may associate the quality data 160 with a pass. If the quality data 160 does not satisfy the threshold quality data 185, the industrial printer may associate the quality data 160 with a fail. It will be appreciated that the vision system 102 could instead compare the quality data 160 to the threshold quality data 185, and transmit the result of the comparison to the industrial printer 104, e.g. pass or fail. In this example, the industrial printer 104 may provide the vision system 102 with the threshold quality data 185, such as during the calibration routine.

[0165] The industrial printer 104 may output, on the HMI 122, an indication that the second image data 172 corresponds with a pass or a fail. The second image data 172 may be used to display an image of the second mark as printed on the HMI 122, alongside an indication of whether the second mark corresponds with a pass or a fail. Whether the industrial printer 104 outputs, on the HMI 122, an indication that the second image data 172 corresponds with a pass or a fail may depend on whether the second image data 172 corresponds with a pass or a fail. For example, the industrial printer 104 may only

[0166] 69532054-3 M&C PM364762GB

[0167] 27 output an indication that the second image data 172 corresponds with a pass or a fail if the second image data 172 corresponds with a fail. Instead of, or in addition to, outputting an indication that the second image data 172 corresponds with a pass or a fail, the industrial printer may output a numerical score obtained from the quality data 160.

[0168] If the second image data 172 corresponds with a fail, the industrial printer 104 may output an alert to the user. Any suitable alert may be used. For example, the alert may be a visual alert on the HMI 122, and / or a sound alert, via a speaker (not shown). The alert may only be output if a predetermined number of fails are recorded. For example, if there are a predetermined number of consecutive fails (such as five), and / or a total number of fails in the print job (such as one hundred), then the alert may be output.

[0169] The threshold quality data 185 may be a default value, or may be set by a user using the HMI 122. For example, when setting up the print job, or during calibration, the user may input an indication of their desired quality such that the corresponding threshold quality data 185 is generated. Figure 5 illustrates an example display 500 that can be output on the HMI 122 and which allows the user to select a desired threshold. The display shown in Figure 5 may be presented to the user after step 208 and before step 400. That is, the following calibration, and prior to starting the full production run, the user may interact with display 500 to select their desired quality threshold.

[0170] The display 500 displays a virtual selector 502 that can be manipulated by the user to select a threshold value 504. In this example, the virtual selector is in the form of a virtual slider 502 configured to be slid along an axis when manipulated by the user. The virtual slider’s position along the axis determines the quality threshold value 504. That is, the state of the virtual selector determines the quality threshold valve 504. In examples where the HMI comprises a touch screen, the user may manipulate the virtual slider by touching and dragging the virtual slider along the axis.

[0171] In the example shown, the quality threshold value 504 takes a value between 1 and 5, with 1 being the least restrictive threshold value and 5 being the most restrictive threshold value. Once selected, corresponding threshold quality data 185 may be generated. For example, in the example shown, if the user selects “3” as the quality threshold value, threshold quality data 185 is determine that corresponds with “3”. This

[0172] 69532054-3 M&C PM364762GB

[0173] 28 may mean scaling the threshold value to a value that can be compared with the quality data 160. In such an example, if the quality data 160 can take a value between 0 and 1 , and a threshold value of “3” is selected, the threshold quality data may be set at 0.6. In this specific example, in order to satisfy the threshold quality data 185, the quality data 160 will then need to correspond with a value of 0.6 or greater. Of course, such scaling may not be required. For example, the threshold value 504 may directly correspond with the values that the quality data 160 can take, e.g. 0 - 1.

[0174] Optionally, when manipulating the slider 502, a simulated image 506 of what the mark will generally look like when the quality is equal to the selected quality threshold value 504 is output on the HMI. The user can then use the simulated image 506 to judge how restrictive the threshold should be. The simulated image may be generated in any suitable way. For example, a model may have been trained on labelled data, the labelled data comprising images of characters and an associated label indicating their quality. The model may take as input the image data of the mark to be printed according to the print design specification 130 and a value for the threshold value 504, and may generate and output the simulated image 506. Alternatively, the simulated image may be selected from a set of pre-generated images of random characters, and / or marks, where each image corresponds with a particular value of quality. For example, if the user wishes to print “ABC”, and selects a quality threshold value 504 of “3”, the industrial printer 104 may select the individual images of the characters “A”, “B” and “C” from the pre-generated images, each image having a quality associated with the quality threshold value 504 of “3”, and may display these characters as a simulated image 506.

[0175] As noted above, following calibration, the quality data 160 associated with the quality of the first printed mark contained within the first image data 170 may also be transmitted to the industrial printer 104. The quality data 160 may be presented to the user on the HMI 122 as the current print quality, which in this example has a value of “5”. The HMI 122 also displays an image 508 of the printed first mark. The HMI 122 may also display comments 510 to the user that provide an indication to the user how the quality may be improved if desired.

[0176] Advantageously, allowing the user to set the quality threshold value 504 provides greater customization. For example, in some applications a relatively high quality for

[0177] 69532054-3 M&C PM364762GB

[0178] 29 the mark may not be required. If the quality threshold value is not customizable, but set at a default level, the system may output that a particular mark has failed inspection that would otherwise be considered acceptable by the user for the particular application.

[0179] While it has been described that generation of the mapping data 180 and reference data 174, 176, takes place at the vision system 102, it will be appreciated that in an alternative implementation, the processing can take place elsewhere. For example, image data 170, 172 may be transferred from the vision system 102 directly to the industrial printer 104, and the industrial printer may generate the mapping data 180 and reference data 174, 176. However, it is advantageous to have the vision system 102 generate the mapping data 180 and reference data 174, 176 as the image data 170, 172 generated by the vision system 102 may have a relatively large file size. Thus, it is beneficial to minimise transmittal of the image data 170, 172 to improve bandwidth.

[0180] The ordering of the steps of the various example methods are described for illustrative purposes only, and it will be appreciated that the ordering may differ to that described. It will further be appreciated that not all steps may be required.

[0181] The techniques described above may be implemented in hardware, firmware, software, or any combination thereof. The techniques may also be implemented as instructions stored on a machine-readable medium, which may be read and executed by one or more processors. A machine-readable medium may include any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computing device). For example, a machine-readable medium may include read only memory (ROM); random access memory (RAM); magnetic storage media; optical storage media; flash memory devices; electrical, optical, acoustical or other forms of propagated signals (e.g. carrier waves, infrared signals, digital signals, etc.), and others. Further, firmware, software, routines, instructions may be described herein as performing certain actions. However, it should be appreciated that such descriptions are merely for convenience and that such actions in fact result from computing devices, processors, controllers, or other devices executing the firmware, software, routines, instructions, etc. and in doing that may cause actuators or other devices to interact with the physical world.

[0182] 69532054-3 M&C PM364762GB

[0183] 30

[0184] While specific embodiments of the invention have been described above, it will be appreciated that the invention may be practiced otherwise than as described. The descriptions above are intended to be illustrative, not limiting. Thus it will be apparent to one skilled in the art that modifications may be made to the invention as described without departing from the spirit and scope of the invention.

[0185] 69532054-3

Claims

M&C PM364762GB31CLAIMS:

1. A computer implemented method comprising: printing, by an industrial printer, a first mark on a first object, the first mark associated with first print data, the first print data comprising coordinates associated with the first mark to be printed in a first coordinate system; capturing, by a vision system, a first image of the printed first mark to generate first image data, the first image data comprising coordinates associated with the first printed mark in a second coordinate system; and generating, based on the first print data and the first image data, mapping data, the mapping data indicating a mapping between the first coordinate system and the second coordinate system.

2. The computer implemented method of claim 1, the method further comprising: generating, based on the first print data and mapping data, reference data associated with the first mark to be printed; training a model using the reference data associated with the first mark to be printed, the trained model configured to process image data generated by the vision system and output quality data associated with a mark contained within the image data.

3. The computer implemented method of claim 2, wherein the reference data associated with the first mark to be printed comprises a synthetic image of the first mark to be printed in the second coordinate system.

4. The computer implemented method of claim 3, wherein generating the reference data associated with the first mark to be printed comprises: generating, using the first print data associated with the first mark to be printed, a synthetic image of the first mark to be printed in the first coordinate system; projecting the synthetic image of the first mark to be printed in the first coordinate system into the second coordinate system using the mapping data to obtain the synthetic image of the first mark to be printed in the second coordinate system.

5. The computer implemented method of any preceding claim, further comprising:69532054-3M&C PM364762GB32 printing, by the industrial printer, a second mark on a second object, the second mark associated with second print data, the second print data comprising coordinates associated with the second mark to be printed in the first coordinate system; generating, based on the second print data and mapping data, reference data associated with the second mark to be printed, the reference data associated with the second mark to be printed comprising coordinates associated with the second mark to be printed in the second coordinate system; capturing, by the vision system, a second image of the second printed mark to generate second image data, the second image data comprising coordinates associated with the second printed mark in the second coordinate system; comparing the reference data associated with the second mark to be printed and the second image data to determine quality data associated with the print of the second mark.

6. The computer implemented method of claim 5, wherein the reference data associated with the second mark to be printed comprises a synthetic image of the second mark to be printed in the second coordinate system.

7. The computer implemented method of claim 6, wherein generating the reference data associated with the second mark to be printed comprises: generating, using the print data associated with the second mark to be printed, a synthetic image of the second mark to be printed in the first coordinate system; projecting the synthetic image of the second mark to be printed in the first coordinate system into the second coordinate system using the mapping data to obtain the synthetic image of the second mark to be printed in the second coordinate system.

8. The computer implemented method of any of claims 5, 6 or 7, when dependent on claim 2, wherein comparing the reference data associated with the second mark to be printed and the second image data to determine quality data associated with the print of the second mark comprises: processing the reference data associated with the second mark to be printed and the second image data using the model to obtain an output from the model; determining the quality data based on the output from the model.69532054-3M&C PM364762GB339. The computer implemented method of claim 8, wherein processing the reference data associated with the second mark to be printed and the second image data using the model to obtain an output from the model further comprises: determining a first set of regions associated with the reference data associated with the second mark to be printed, the first set of regions corresponding to bounding boxes associated with the second mark to be printed; determining a second set of regions associated with the second image data, the second set of regions corresponding to bounding boxes associated with the printed second mark; processing the first regions and second regions using the model to obtain the output from the model.

10. The computer implemented method of any of claims 5 to 9, further comprising: comparing the quality data to a quality threshold; determining, based on the comparing, whether the quality of the second mark satisfies the quality threshold.

11. The computer implemented method of claim 10, further comprising: displaying a virtual selector on a human machine interface associated with the industrial printer, the virtual selector configured to allow a selection, via user input to the human machine interface, of the quality threshold; determining the quality threshold based on a state of the virtual selector.

12. The computer implemented method of claim 11 , further comprising: receiving user input at the human machine interface, the user input manipulating the virtual selector to change the state of the virtual selector.

13. The computer implemented method of claim 11 or 12, wherein the virtual selector comprises a virtual slider configured to be slid along an axis when manipulated by the user, wherein the state of the virtual selector comprises the virtual slider’s position along the axis, and wherein the virtual slider’s position along the axis determines the quality threshold.69532054-3M&C PM364762GB3414. The computer implemented method of claims 10 to 13, further comprising displaying on the human machine interface a predicted image of a mark having a quality associated with the determined quality threshold.

15. The computer implemented method of claims 11 to 14, wherein the human machine interface is coupled to the industrial printer.

16. The computer implemented method of any of claims 5 to 15, wherein the first mark and second mark are associated with the same print design specification.

17. The computer implemented method of any preceding claim, wherein the first and second coordinate systems are two dimensional coordinate systems.

18. The computer implemented method of any preceding claim, wherein the mapping data comprises a transformation matrix.

19. A system comprising an industrial printer and a vision system; the industrial printer configured to: print a first mark on a first object, the first mark associated with first print data, the first print data comprising coordinates associated with the first mark to be printed in a first coordinate system; transmit the first print data to the vision system; print a second mark on a second object, the second mark associated with second print data, the second print data comprising coordinates associated with the second mark to be printed in the first coordinate system; transmit the second print data to the vision system; the vision system configured to: capture a first image of the printed first mark to generate first image data, the first image data comprising coordinates associated with the first printed mark in a second coordinate system; generate, based on the first print data and the first image data, mapping data, the mapping data indicating a mapping between the first coordinate system and the second coordinate system; generate, based on the second print data and mapping data, reference data associated with the second mark to be printed, the reference data69532054-3M&C PM364762GB35 associated with the second mark to be printed comprising coordinates associated with the second mark to be printed in the second coordinate system; capture a second image of the second printed mark to generate second image data, the second image data comprising coordinates associated with the second printed mark in the second coordinate system; compare, using a model, the reference data associated with the second mark to be printed and the second image data to determine quality data associated with the print of the second mark.

20. A computer program comprising instructions, which when the program is executed by an industrial printer, cause the industrial printer to: print a first mark on a first object, the first mark associated with first print data, the first print data comprising coordinates associated with the first mark to be printed in a first coordinate system; transmit the first print data to a vision system; print a second mark on a second object, the second mark associated with second print data, the second print data comprising coordinates associated with the second mark to be printed in the first coordinate system; transmit the second print data to the vision system; and which when the program is executed by a vision system, cause the vision system to: capture a first image of the printed first mark to generate first image data, the first image data comprising coordinates associated with the first printed mark in a second coordinate system; generate, based on the first print data and the first image data, mapping data, the mapping data indicating a mapping between the first coordinate system and the second coordinate system; generate, based on the second print data and mapping data, reference data associated with the second mark to be printed, the reference data associated with the second mark to be printed comprising coordinates associated with the second mark to be printed in the second coordinate system; capture a second image of the second printed mark to generate second image data, the second image data comprising coordinates associated with the second printed mark in the second coordinate system;69532054-3M&C PM364762GB36 compare, using a model, the reference data associated with the second mark to be printed and the second image data to determine quality data associated with the print of the second mark.

21. A computer implemented method comprising: displaying a virtual selector on a human machine interface associated with an industrial printer, the virtual selector configured to allow a selection, via user input to the human machine interface, of a quality threshold associated with a print of a mark by the industrial printer; determining a quality threshold associated with the print of the mark by the industrial printer based on a state of the virtual selector.

22. The computer implemented method of claim 21 , further comprising: receiving user input at the human machine interface, the user input manipulating the virtual selector to change the state of the virtual selector.

23. The computer implemented method of claim 21 or 22, wherein the virtual selector comprises a virtual slider configured to be slid along an axis when manipulated by the user, wherein the state of the virtual selector comprises the virtual slider’s position along the axis, and wherein the virtual slider’s position along the axis determines the quality threshold.

24. The computer implemented method of claims 21 , 22 or 23, further comprising displaying on the human machine interface a predicted image of the mark having a quality associated with the determined quality threshold.

25. The computer implemented method of claims 21 to 24, wherein when displaying the virtual selector on the human machine interface, and prior to receiving user input, the quality threshold comprises a default value.69532054-3

Images