Methods and systems for monitoring assembly lines for medical devices

Abstract

Described herein are analytic techniques for identifying irregularities occurring in an assembly line that have a high impact on the rate at which medical devices are rejected. These techniques involve the computation of scores indicative of the extent to which occurrence of different types of alarms occurring during manufacturing are root causes responsible for different types of anomalies exhibited by the manufactured products. Scores determined using these techniques provide a clear indication as to which types of irregularities in the manufacturing machinery have a greater impact on the rate at which medical devices are ultimately rejected. This information is insightful because it enables operators to perform targeted maintenance on the manufacturing line. Addressing irregularities that produce high scores has a greater chance to reduce the rejection rate than addressing irregularities that produce low scores. Thus, these techniques may be viewed as being able to identify the root cause responsible for the rejection of manufactured products.

Description

METHODS AND SYSTEMS FOR MONITORING ASSEMBLY LINES FOR MEDICAL DEVICESFIELD OF THE DISCLOSURE

[0001] The techniques described herein are generally related to methods and systems for monitoring assembly lines for medical devices. For example, some embodiments described herein are related to methods and systems for identifying irregularities occurring in an assembly line that have a high impact on the rate at which medical devices are ultimately rejected.BACKGROUND OF THE DISCLOSURE

[0002] In a production assembly line, defect detection systems are used to automatically detect defects in manufactured components that move through the assembly line. Some defect detection systems capture an image of a manufactured component while the manufactured component enters, is processed through, or exits the assembly line and process the captured image to determine whether the image indicates a defect in the manufactured component.SUMMARY OF THE DISCLOSURE

[0003] Some embodiments relate to manufacturing system, comprising manufacturing machinery configured to output a plurality of batches of medical devices; a plurality of sensors configured to generate alarm data representing incidence of alarms resulting from irregular operation of the manufacturing machinery during manufacturing of each batch of medical devices, wherein each alarm belongs to one of a plurality of alarm types, and each alarm type is associated with a different type of irregular operation of the manufacturing machinery; one or more cameras configured to visually inspect each medical device output by said manufacturing machinery and determine whether to reject said medical device as being compromised due to a detected type of anomaly out of one or more possible types of anomalies; and at least one computer processor and computer memory storing processorexecutable instructions that, when executed by the at least one computer processor, cause the at least one computer processor to: generate system effectiveness data for each batch of medical devices output by the manufacturing machinery, wherein the system effectiveness data is calculated based at least in part on a rate at which medical devices in said batch were rejected, generate a function that fits the system effectiveness data relative to the alarm data, and generate score data using the function, the score data comprising, for each alarm type ofthe plurality of alarm types, an associated score representing an extent to which incidence of alarms of said alarm type is a root cause responsible for the detected type of anomaly.

[0004] Some embodiments relate to a manufacturing system, comprising: manufacturing machinery configured to output a plurality of batches of medical devices: one or more cameras configured to visually inspect each medical device output by said manufacturing machinery and determine whether to reject said medical device as being compromised due to a detected type of anomaly out of one or more possible types of anomalies; and at least one computer processor and computer memory storing processor-executable instructions that, when executed by the at least one computer processor, cause the at least one computer processor to: generate system effectiveness data for each batch of medical devices output by the manufacturing machinery, wherein the system effectiveness data is calculated based at least in part on a rate at which medical devices in said batch were rejected, receive genealogy data representing genealogy information associated with the batches of medical devices, generate a function that fits the system effectiveness data relative to the genealogy data, and generate score data using the function, the score data comprising a score representing an extent to which the genealogy information associated with the batches of medical devices is a root cause responsible for the generated system effectiveness data.

[0005] Some embodiments relate to a method for monitoring a manufacturing system, comprising: receiving, from a plurality of sensors, alarm data representing incidence of alarms resulting from irregular operation of manufacturing machinery, wherein the manufacturing machinery is configured to output a plurality of batches of medical devices, wherein each alarm belongs to one of a plurality of alarm types, and each alarm type is associated with a different type of irregular operation of the manufacturing machinery; receiving information indicative of whether a medical device of the plurality of batches of medical devices is to be rejected as being compromised due to a type of anomaly out of one or more possible types of anomalies detected by one or more cameras; generating system effectiveness data for each batch of medical devices output by the manufacturing machinery, wherein the system effectiveness data is calculated based at least in part on a rate at which medical devices in said batch were rejected; generating a function that fits the system effectiveness data relative to the alarm data; and generating score data using the function, the score data comprising, for each alarm type of the plurality of alarm types, an associated score representing an extent to which incidence of alarms of said alarm type is a root cause responsible for the generated system effectiveness data.

[0006] Some embodiments relate to a method for monitoring a manufacturing system, comprising: receiving information indicative of whether a medical device of a plurality of batches of medical devices output by manufacturing machinery is to be rejected as being compromised due to a type of anomaly out of one or more possible types of anomalies detected by one or more cameras; generating system effectiveness data for each batch of medical device, wherein the system effectiveness data is calculated based at least in part on a rate at which medical devices in said batch were rejected; receiving genealogy data representing genealogy information associated with the batches of medical devices; generating a function that fits the system effectiveness data relative to the genealogy data, and generating score data using the function, the score data comprising a score representing an extent to which the genealogy information associated with the batches of medical devices is a root cause responsible for the generated system effectiveness data.

[0007]

[0008] There has thus been outlined, rather broadly, the features of the disclosed subject matter in order that the detailed description thereof that follows may be better understood, and in order that the present contribution to the art may be better appreciated. There are, of course, additional features of the disclosed subject matter that will be described hereinafter and which will form the subject matter of the claims appended hereto. It is to be understood that the phraseology and terminology employed herein are for the purpose of description and should not be regarded as limiting.BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Various objectives, features, and advantages of the disclosed subject matter can be more fully appreciated with reference to the following detailed description of the disclosed subject matter when considered in connection with the following drawings, in which like reference numerals identify like elements.

[0010] FIG. 1 is a block diagram showing a manufacturing system for medical devices, according to some embodiments.

[0011] FIG. 2 is a block diagram showing a portion of the system of FIG. 1 in additional detail, according to some embodiments.

[0012] FIG. 3A is a flow chart showing an exemplary computerized method for monitoring a manufacturing system for medical devices, according to some embodiments.

[0013] FIG. 3B is another flow chart showing another exemplary computerized method for monitoring a manufacturing system for medical devices, according to some embodiments.

[0014] FIG. 4 is a graphical user interface showing various panels enabling users to select subsets of batches of medical devices, according to some embodiments.

[0015] FIG. 5 shows an exemplary subset of batches selected by a user using one of the panels of FIG. 4, according to some embodiments.

[0016] FIG. 6A is another graphical user interface showing a table including score data associated with the selected subset of batches of FIG. 5, according to some embodiments.

[0017] FIG. 6B is another graphical user interface showing another table including score data associated with the selected subset of batches of FIG. 5, according to some embodiments.

[0018] FIG. 6C is another graphical user interface showing another table including score data associated with the selected subset of batches of FIG. 5, according to some embodiments.

[0019] FIG. 7 shows an illustrative implementation of a computer system that may be used to perform any of the aspects of the techniques and embodiments disclosed herein, according to some embodiments.DETAILED DESCRIPTION OF THE DISCLOSURE

[0020] The techniques described herein provide for methods and systems for identifying irregularities occurring in pharmaceutical assembly lines that have a high impact on the rate at which medical devices are ultimately rejected.

[0021] Pharmaceutical assembly lines for assembling medical devices (e.g., for filling syringes with medicine, assembling syringes into medication delivery devices, or packaging such devices) can suffer from various irregularities that may lead to the rejection of a significant fraction of the manufactured products. Products may be non-compliant for a variety reasons, such as due to particulate contamination, fill volume inconsistencies, seal integrity issues, defects in syringe components, manufacturing variations, etc. To minimize these irregularities, manufacturers implement stringent quality control measures and continuous monitoring.

[0022] The inventors have recognized and appreciated that conventional approaches to quality control and monitoring face several challenges. Some pharmaceutical assembly linesrely on sensors designed to raise alarms responsive to the occurrence of irregularities in the line and inspection systems to detect and reject defective syringes. The high-volume nature of certain pharmaceutical assembly lines, coupled with the highly sophisticated nature of the machinery involved, can cause each sensor to raise tens or hundreds of alarms each day. Assembly lines often employ hundreds of sensors. Consequently, several thousands of alarm signals may be produced each day. Conventional approaches are based on manual statistical analysis of alarm signals, which can take several weeks or even months to complete. The analysis often requires involvement of technical experts, line engineers and process automation engineers. As a further drawback, conventional approaches are often prone to false positives; they tend to over-reject manufactured products. Over-rejection can increase manufacturing costs significantly.

[0023] Recognizing the limitations discussed above and other issues, the inventors have developed analytic techniques configured to identify irregularities occurring in a manufacturing system that have a high impact on the overall effectiveness of the manufacturing system. Here, the “overall effectiveness of the manufacturing system’’ can be summarized or quantified using system effectiveness data that is calculated based at least in part on a rate at which medical devices are rejected. This system effectiveness data (which, in some embodiments, may take the form of a numerical score) may be calculated anew for each batch of medical devices output by the manufacturing system, such that changes in the manufacturing system’s effectiveness between batches of medical devices may be captured and analyzed. In some embodiments, the system effectiveness data for a batch of medical devices may be calculated based on a rate at which medical devices in said batch were rejected due to detection of one specific type of anomaly out of a plurality of possible types of anomalies. In other embodiments, the system effectiveness data for a batch of medical devices may be calculated based on a rate at which medical devices in said batch were rejected due to detection of any type of anomaly out of the plurality of possible types of anomalies. One specific example of the latter embodiments includes calculation of an “Overall Batch Effectiveness” score, or OBE score, for each batch of medical devices produced. Equation 1 below summarizes one exemplary (and non-limiting) way to calculate an OBE score:a. No. of good units = the number of non-defective medical devices produced by the manufacturing equipment during said batchb. Batch run time = the total amount of time taken to produce said batch of medical devicesc. Rate = the number of devices the manufacturing system is configured or set up to produce per unit time (e.g., per minute)

[0026] The techniques developed by the inventors and described herein involve the computation of metrics (e.g., scores) indicative of the extent to which different types of alarms occurring during manufacturing are root causes responsible for, or which most strongly influence, the overall effectiveness of the manufacturing system. Scores determined using the techniques described herein provide a clear indication as to which types of irregularities in the manufacturing machinery have a greater impact on the rate at which medical devices are ultimately rejected. This information is insightful because it enables operators to perform targeted maintenance on the manufacturing line. Addressing irregularities that produce high scores has a greater chance to reduce the rejection rate than addressing irregularities that produce low scores. In that regard, the techniques described herein may be viewed as being able to identify the root cause responsible for the rejection of manufactured products.

[0027] For example, some embodiments compute a score representing the extent to which the occurrence of an alarm produced by a sensor that detects spilling of liquid in the assembly line is a root cause responsible for increased rejection rates due to detection of a specific type of anomaly associated with insufficient liquid level in the syringes of a batch. If the computed score has a relatively large value, this may indicate that the root cause (or at least one root cause) responsible for the rejection of certain batches due to insufficient liquid level is attributable to the machine that dispenses liquid. Some embodiments generate several scores correlating different types of irregularities in the assembly line (e.g., spilled liquid, insufficient actuation force, inadvertent opening of guard door, inaccurate positioning of intermediate product, etc.) with different types of anomalies in the manufactured products (e.g., insufficient level of medicine, presence of foreign particles, missing needle or plunger, etc.). Other embodiments generate several scores correlating different types of irregularities in the assembly line with rejection of the manufactured products due to any type of anomaly, without regard to any specific type of anomaly. In all of these embodiments, these generated scores enable operators to assess which types of irregularities significantly impact the overalleffectiveness of the manufacturing system and, consequently, to perform targeted maintenance on the manufacturing line. As a result, over-rejection is substantially reduced compared to conventional statistical tools. Additionally, the real-time nature of the techniques described herein represents a significant improvement over the long lead times associated with conventional approaches.

[0028] Accordingly, some embodiments relate to a manufacturing system comprising manufacturing machinery, a plurality of sensors, one or more cameras and a computer system. The manufacturing machinery is configured to output a plurality of batches of medical devices. The sensors are configured to generate alarm data representing incidence of alarms resulting from irregular operation of the manufacturing machinery during manufacturing of each batch of medical devices. Each alarm belongs to one of a plurality of alarm types, and each alarm type is associated with a different type of irregular operation of the manufacturing machinery. The camera(s) visually inspect each medical device output by the manufacturing machinery and determine whether to reject the medical device as being compromised due to a detected type of anomaly. The computer system generates system effectiveness data that is calculated based at least in part on a rate at which medical devices in the batch were rejected. As explained previously, this system effectiveness data may be specific to a particular type of detected anomaly, or it may capture rejections due to any type of anomaly. Further, the computer system generates a function that fits the system effectiveness data with respect to the alarm data. Further, the computer system generates score data using the function, where the score data comprises, for each alarm type of the plurality of alarm types, a score representing the extent to which incidence of alarms of the alarm type is a root cause responsible for, or which strongly influences, the system effectiveness data.

[0029] The inventors have further recognized and appreciated that products of a certain origin may be more susceptible to anomalies than products of a different origin. For example, syringes supplied by a certain vendor may be particularly susceptible to particle contamination. Isolating syringes supplied by that vendor as the root cause responsible for the presence of foreign particles is not straightforward, especially in an assembly line where several different types of irregularities can occur simultaneously. Further techniques developed by the inventors and described herein involve the computation of metrics (e.g., scores) indicative of the extent to which the genealogy associated with batches of products is a root cause responsible for, or which strongly influences, the overall effectiveness of themanufacturing system. Genealogy data may include for example the identity of a vendor, the material used to manufacture a product and / or the country of manufacturing. Scores determined using the techniques described herein provide a clear indication as to which aspects of the genealogy of input materials have a greater impact on the rate at which medical devices are ultimately rejected. This information is insightful because it enables operators to assess whether a particular group of medical devices should be discarded a priori, and / or whether to switch suppliers of raw materials and / or intermediate components.

[0030] Accordingly, some embodiments relate to a manufacturing system comprising manufacturing machinery, one or more cameras and a computer system. The manufacturing machinery is configured to output a plurality of batches of medical devices. The camera(s) visually inspect each medical device output by the manufacturing machinery and determine whether to reject the medical device as being compromised due to a detected type of anomaly. The computer system generates system effectiveness data for each batch of medical devices, wherein the system effectiveness data is calculated based at least in part on a rate at which medical devices in the batch were rejected (whether due to a specific type of anomaly, or due to any anomaly). Further, the computer system receives genealogy data representing genealogy information associated with the batches of medical devices, and generates a function that fits the system effectiveness data with respect to the genealogy data. Further, the computer system generates score data comprising a score representing the extent to which the genealogy of a batch is a root cause responsible for the system effectiveness data.

[0031] In some embodiments, the computation of the scores described above may be limited to a subset of the batches output by the assembly line. The selected subset may correspond to batches pre-classified as being anomalous. Anomalous batches may include, for example, batches exhibiting particularly severe rejection rates. In this way, the techniques described herein can pre-isolate those batches that are most likely to produce high degrees of correlation, leading to a substantial reduction in computational load.

[0032] It should be noted that the techniques developed by the inventors and described herein can be applied to a variety of manufacturing systems, and are not limited to assembly lines for filling syringes with medicine. These techniques may be applied to other types of assembly lines, in pharmaceutical settings as well as in other settings (e.g., industrial).Furthermore, the term “manufacturing” as used herein should be interpreted broadly to include not only making, assembling, or creating items, but also inspecting, processing, sorting, and / or packaging items.

[0033] In the following description, numerous specific details are set forth regarding the systems and methods of the disclosed subject matter and the environment in which such systems and methods may operate, etc., in order to provide a thorough understanding of the disclosed subject matter. In addition, it will be understood that the examples provided below are exemplary, and that it is contemplated that there are other systems and methods that are within the scope of the disclosed subject matter.

[0034] FIG. 1 is a block diagram showing a manufacturing system for medical devices, according to some embodiments. Manufacturing system 100 may include an assembly line configured to output medical devices. A manufactured medical device may include a syringe, a cartridge, a tube, a container, a glass vial, a bottle, ajar, and / or any suitable component. Manufacturing system 100 includes manufacturing machinery 108, which in turn may include one or more pieces of equipment that operate on different segments of the assembly line. In one example, manufacturing machinery 108 is configured to fill syringes with medicine, e.g., liquid medicine. It should be noted that the techniques described herein are not limited to filling syringes with liquid medicine but may be used in conjunction with other types of manufacturing -related operations (e.g., assembling or packaging), other types of medical devices and / or other types of medicines.

[0035] Machinery 108 may receive as input multiple empty syringes. The syringes are empty in that they are not pre-filled with medicine or are partially pre-filled with a substance but have a container with enough available space to receive medicine. Machinery 108 may include equipment configured to fill empty syringes with medicine, whether in batches or individually. In some embodiments, machinery 108 may include a conveyor belt configured to carry syringes through the various stages of the assembly line. Robotic arms may be used for precise handling of syringes throughout the assembly line, reducing human contact and maintaining sterility. Machinery 108 may include equipment that dispenses and accurately fills syringes with liquid medicine. To dispenses medicine, the equipment may include pumps and / or piston mechanisms. In some embodiments, machinery 108 includes equipment configured to insert rubber stoppers into the filled syringes to seal them. Additional equipment may be used to attach caps to the syringes and crimp them to secure the stoppers in place. In some embodiments, machinery 108 includes equipment configured to apply labels to the syringes with information such as dosage, expiration date, and batch identifier. In some embodiments, machinery 108 includes equipment configured to package the filled and sealed syringes into boxes or other types of packaging for distribution.

[0036] Manufacturing machinery 108 outputs medical devices, which may be organized in batches in some embodiments. For example, machinery 108 may output batches of syringes filled with medicine. Each batch may include any suitable number of medical devices, such as between 10 medical devices and 10,000 medical devices, between 10 medical devices and 1,000 medical devices, between 10 medical devices and 100 medical devices, between 100 medical devices and 10,000 medical devices, between 100 medical devices and 1,000 medical devices or between 1,000 medical devices and 10,000 medical devices, just to name a few exemplary ranges. The batch size (in terms of the number of medical devices contained in a batch) may be constant across the batches, or may vary. Each batch may be ultimately packaged in a container for further processing (e.g., shipping).

[0037] Manufacturing system 100 includes multiple sensors (e.g., 110i, 1 IO2...110N) configured to monitor the assembly of medical devices by the manufacturing machinery. For example, the sensors 110 may be configured to generate alarm data representing incidence of alarms resulting from irregular operation of manufacturing machinery 108 during manufacturing of each batch of medical devices. A sensor raises an alarm when an irregular operation is detected. The sensors may be embedded within manufacturing machinery 108. The alarms may be of different types, depending on the type of sensor that generated them. In one example, one type of sensor includes circuitry configured to detect that an actuator has malfunctioned and / or circuitry configured to detect that an actuator is not exerting sufficient force to push a syringe into the right position. In another example, one type of sensor includes a camera (and related software) configured to detect spilling or dripping of liquid medicine on the line. In another example, one type of sensor includes circuitry configured to identify whether an intermediate product progressing through the line does not satisfy certain parameters, such as size, width, height, weight requirements. In another example, one type of sensor includes circuitry configured to determine whether the guard door of a piece of machinery has been inadvertently opened. In another example, one type of sensor includes circuitry configured to monitor the position of a syringe and to raise an alarm if it determines that the position has deviated from a predefined trajectory. In another example, one type of sensor includes circuitry configured to determine whether a jam has occurred within manufacturing machinery 108. In another example, one type of sensor includes circuitry configured to determine whether a piece of equipment within manufacturing machinery 108 has unexpectedly stopped its operation and / or has produced a warning. Other types of sensors are possible.

[0038] In some embodiments, there may be several types of sensors (e.g., dozens), and consequently, several types of alarms as each type of alarm is associated with a particular type of sensor. Depending on the integrity of the operation, each sensor may generate tens, hundreds or even thousands of alarms each day. As such, the sensors may generate vast amounts of data. In some embodiments, manufacturing system 100 may be organized in stages. For example, a first stage may involve pre -filling medical devices with some initial amount of medicine, a second stage may involve inspecting that the medical devices have been properly pre-filled, and a third stage may involve filling the medical devices with the desired amount of medicine. In some embodiments, the sensors (e.g., 1101, 1 IO2...110N) may be deployed to monitor the operations of manufacturing system 100 in connection with the first and second stages. Monitoring of the third stage may be optional. However, not all embodiments are limited in this respect.

[0039] Inspection machinery 120 inspects each medical device output by manufacturing machinery 120 and determines whether to reject the medical device as being compromised due to a detected type of anomaly. In some embodiments, inspection machinery 120 may only flag a medical device (or batch of medical devices) for review by a human operator, and it may ultimately be the operator’s responsibility to decide whether to actually reject a product. Inspection machinery 120 may be positioned at a stage of the assembly line that precedes a packaging step. In some embodiments, inspection machinery 120 includes one or more cameras 121, as shown in FIG. 2. Thus, the inspection performed by inspection machinery 120 may be performed visually. In one example, multiple cameras may be used to inspect multiple medical devices (e.g., in a batch) in parallel. A software module embedded in inspection machinery 120 may be programmed (e.g., trained) to use signals produced by the cameras to determine whether to reject the medical device as being compromised. The software module may recognize whether a syringe is scratched, cracked or contaminated. Syringes that are damaged in some of these respects may be rejected. Additionally or alternatively, the software module may measure the liquid level in each syringe, and may compare the fill level against predefined standards to ensure consistency. Syringes with insufficient fill level may be rejected.

[0040] Inspection machinery 120 may include other types of inspection sensors (123) in addition to or in lieu of camera(s) 121. In one example, an inspection sensor 123 includes a laser-based system that uses lasers to detect particles within the liquid. As the laser beam passes through the syringe, any particles present in the liquid will scatter the light, which isthen detected by photodetectors. Tn another example, an inspection sensor 123 includes an electro-optical system that can differentiate between different types of particles based on their optical properties. Syringes having certain types of particles and / or particles in excess of a certain amount may be rejected. Additionally or alternatively, an inspection sensor 123 may include a dimension sensor, e.g., a sensor configured to measure the dimension of a medical device, such as the height and / or width and / or depth and / or shape of a medical device or a portion of a medical device. Examples of dimension sensors include laser displacement sensors and more generally optical sensors, ultrasonic sensors, triangulation sensors, contact sensors (e.g., micrometers and calipers), scanners, etc. Additionally or alternatively, an inspection sensor 123 may include a weight sensor, e.g., a sensor configured to measure the weight of a medical device. Examples of weight sensors include scales and load cells, such as strain gauge load cells, hydraulic load cells, pneumatic load cells, capacitive load cells, piezoelectric load cells, etc. Additionally or alternatively, an inspection sensor 123 may include a compliance sensor, e.g., a sensor configured to assess whether a medical device is operating as intended. Examples of compliance sensors include sensors configured to determine whether a particular portion of a medical device (e.g., an infeed, an outfeed, a gripper, a plunger, a chute, etc.) is working properly. Additionally or alternatively, an inspection sensor 123 may include a mechanical sensor, e.g., a sensor configured to perform a measurement with respect to a particular mechanical characteristic of a medical device, such as pressure, strength, stress, compressibility, etc.

[0041] Medical devices may be rejected or accepted individually or in batches. Inspection machinery 120 may reject medical devices for any among different types of anomalies (e.g., 122i, 1222...122pin FIG. 2). As noted above, one type of anomaly may indicate insufficient fill level, another type of anomaly may indicate that a syringe is damaged (e.g., scratched or cracked) and another type of anomaly may indicate the presence of foreign particles in a syringe. Other types of anomalies are also possible, including an anomaly indicating a missing or damaged needle, missing or damaged plunger and damaged glass sidewall, among others. As further shown in FIG. 2, inspection machinery 120 may pass or fail a medical device over each type of anomaly. For example, a medical device that has an excessive amount of foreign particles but is compliant in every other respect may fail the type of anomaly associated with the presence of foreign particles but may pass the other types of anomalies. In some embodiments, a medical device is to be rejected if it fails at least one type of anomaly.

[0042] Referring back to FIG. 1 , a rejection unit 125 may be configured to generate system effectiveness for each batch of medical device, calculated based at least in part on a rate at which medical devices in that batch were rejected (either due to a specific type of anomaly, or due to any anomaly). For example, the rejection data may indicate that one medical device per batch has been rejected due to a first type of anomaly (e.g., missing needle) and that three medical devices per batch have been rejected due to a second type of anomaly (e.g., presence of foreign particles). Rejection unit 125 may be implemented with at least one computer processor and computer memory storing processor-executable instructions that, when executed by the at least one computer processor, cause the at least one computer processor to generate the rejection data.

[0043] The system of FIG. 1 further includes a classification unit 130, a regression unit 140 and a score unit 160. It should be noted that classification unit 130 may be omitted in some embodiments. As described in connection with rejection unit 125, classification unit 130, regression unit 140 and score unit 160 may be implemented with at least one computer processor and computer memory storing processor-executable instructions that, when executed by the at least one computer processor, cause the at least one computer processor to perform the steps described below. In some embodiments, a common processor or set of processors is shared among rejection unit 125, classification unit 130, regression unit 140 and score unit 160. In other embodiments, each unit has one or more dedicated processors.

[0044] Classification unit 130 is configured to classify batches of medical devices depending on the rejection data generated by rejection unit 125. The classification may be performed using different types of classifiers, including for example a binary classifier 132 and / or a severity ranking classifier 134. Exemplary implementations of these classifiers are described in detail further below. Batches may be classified as anomalous or non-anomalous. Based on the classification, a batch selector 136 may select a subset of the batches inspected by inspection machinery 120. In some embodiments, only the anomalous batches are selected. Thus, only the identifiers associated with the anomalous batches are provided as input to score unit 160. The identifiers of batches that are not classified as anomalous may not be provided as input to score unit 160. As such, score unit 160 may only consider the anomalous batches in some embodiments.

[0045] As noted above, in some embodiments, classification unit 130 may be omitted. In these embodiments, the identifiers of all the batches inspected by inspection machinery 120 may be considered by score unit 160. Alternatively, only the identifiers of a predefined set ofbatches may be considered by score unit 160, where the predefined batches may be selected based on parameters other than the rejection data generated by rejection unit 125. In another example, a subset of the batches may be selected randomly or via manual user input.

[0046] Regression unit 140 includes an anomaly regressor 142 and a genealogy regressor 144. Anomaly regressor 142 is configured to generate a function that describes the relationship between system effectiveness data produced by rejection unit 125 and alarm data produced by sensors 110. For example, anomaly regressor 142 may aim to identify a mathematical function / that fits the observed system effectiveness data relative to the alarm data. Anomaly regressor 142 may be implemented in any of numerous ways, examples of which are described in detail further below. Similarly, genealogy regressor 144 is configured to generate a function that describes the relationship between system effectiveness data produced by rejection unit 125 and genealogy data produced by a genealogy data unit 150.

[0047] The genealogy data unit 150 may include a storage unit storing the genealogy data.The genealogy data may represent various types of information associated with a particular batch, including for example the identity of the vendor that supplied empty syringes, the composition (e.g., the material(s) of which the syringes are made), the country and / or manufacturing facility in which the syringes are manufactured, etc. Genealogy regressor 144 may aim to identify a mathematical function g that fits the observed system effectiveness data relative to the genealogy data.

[0048] Anomaly regressor 142 and genealogy regressor 144 may be implemented in any of numerous ways. For example, a regressor may be implemented as a tree-based light gradient boosting machine (LightGBM) regressor - a gradient boosting framework that uses treebased learning algorithms. In tree-based learning, the individual models used in gradient boosting are decision trees. A decision tree splits the data into subsets based on feature values, aiming to maximize the separation of the target variable within the subsets.LightGBM is a specific implementation of gradient boosting that is designed to be highly efficient and scalable. LightGBM uses a histogram-based method for finding the best split points, which speeds up the training process. A LightGBM regressor builds an ensemble of decision trees to determine correlations. The model may be trained with labeled data associated with historical batches. Alternative implementations for regressors 142 and 144 include linear regressors, polynomial regressors, ridge regressors, Lasso regressors, neural networks, etc.

[0049] In some embodiments, a function produced by a regressor (whether function / or g) may be used to output metrics indicating how well the mathematical function fits the observed data (e.g., root mean square error (RMSE) and / or R2). Correlations are evaluated to assess the likelihood that a particular type of anomaly is responsible for the system effectiveness data calculated over a number of batches. For example, an alarm type having a correlation (r) less than a certain threshold (e.g., 0.1) may be discarded from the list of potential candidates. Vice versa, an alarm type that is highly correlated (e.g., r > 0.8) may be highlighted for the operator' s attention.

[0050] Some of the regressors may be highly correlated with each other. For example, a particular pair of alarm types may be highly correlated with each other. Consequently, whenever an alarm of the first type occurs, an alarm of the second type is also very likely to occur. In such cases, a regression may be performed using only one of the alarm types of the pair (e.g., the alarm type with the higher incidence rate). The other alarm type may be excluded from the regression to not cause misleading results.

[0051] Score unit 160 receives as input the function(s) produced by regression unit 140 and, optionally, the identifiers associated with the anomalous batches selected by batch selector 136. Based on the received input, score unit 160 generates score data that indicate either the extent to which the incidence of a type of alarm is a root cause responsible for the observed system effectiveness data or the extent to which the genealogy of a batch is a root cause responsible for the observed system effectiveness data (or both). The root cause may be viewed as the primary source (or at least as an underlying factor) that explains the anomalies identified by rejection unit 125. As such, score data may provide an indication as to the origin of anomalous characteristics in the medical devices produces by manufacturing system 100. Identifying the root cause helps in understanding the true nature of relationships within the data, allowing for more accurate predictions, better decision-making, and effective interventions or solutions.

[0052] In one example, the score data may indicate whether the occurrence of an alarm produced by a particular type of sensor (e.g., a sensor that detects spilling of liquid in the assembly line) is a root cause responsible for rejection of medical products due to detection of a particular type of anomaly (e.g., insufficient liquid level in the syringes of a batch). Where this score has a relatively large value, that may indicate that the root cause (or at least one root cause) responsible for the rejection of certain batches is attributable to the machine that dispenses liquid, for example. The score data may include a matrix of scores where eachelement of the matrix is indicative of an estimated strength of the causal relationship between a particular type of alarm and a particular type of detected anomaly. Addressing the irregularity that gave rise to alarms of a particular type that is highly correlated with occurrences of a certain type of anomaly may help reduce the rate at which syringe are being rejected.

[0053] In another example, score data may indicate whether a parameter stored in the genealogy data unit 150 (e.g., the fact that certain raw materials or intermediate components, such as syringes, are supplied by a particular vendor, or manufactured by a particular manufacturing plant) is a root cause responsible for a rejection rate associated with a particular type of anomaly (e.g., excessive variability in the parts of medical devices). Where this score has a relatively large value, that may indicate that the root cause (or at least one root cause) responsible for the rejection of certain batches is attributable to the abovereferenced store parameter in the genealogy data unit 150 (e.g., variability in the syringes supplied by that certain vendor). The score data may include a matrix of scores where each element of the matrix represents the causal relationship between a particular type of genealogy information and a particular type of detected anomaly. Removing raw materials or intermediate components of a certain genealogy that is highly correlated with occurrences of a certain type of anomaly may help reduce the rate at which syringe are being rejected.

[0054] In embodiments that include classification unit 130, score unit 160 generates score data across the batches that have been classified as anomalous. Batches that have not been classified as anomalous are not considered in the score generation. This reduces the computation involved in the generation of score data significantly. In other embodiments, score unit 160 generates score data across all the batches that have been inspected by inspection machinery 120 or across a predefined set of batches (e.g., selected randomly, manually by a user, or using criteria other than the classification discussed above).

[0055] Score unit 160 may be implemented in any of numerous ways. In one example, score unit 160 may use Shapley Additive Explanations to determine the degree to which each alarm type predicts the incidence of a particular type of anomaly. As such, the score data may include Shapley scores. First, regressors may be organized in a random order. Subsequently, a first regression is run using only the first variable in the ordered list, and the RMSE is computed. Then, a second regression is run by adding the subsequent regressor in the ordered list, and the RMSE is computed one more time. The Shapley value for the second added variable is represented by the difference in the RMSEs when the second variable is added.Then, a third regression is run by adding the subsequent regressor in the ordered list, and the RMSE is computed one more time. This time, the Shapley value for the third added variable is the difference in the RMSEs when the third variable is added. The algorithm proceeds iteratively in this way. The variables having the highest Shapley values represent the root causes having the greatest impact in predicting defect rates.

[0056] It should be noted that the Shapley values computed in the manner discussed above may depend on the original order of the regressors. To reduce the impact of this limitation, this process may be repeated for every possible ordering of the regressor variables while determining the Shapley value for each regressor variable. At the end of the process, N Shapley values are obtained for each regressor variable, where N is the total number of ways in which regressor variables can be ordered. For each regressor variable, the average Shapley value may be computed by averaging the N Shapley values at the end of the process. The average Shapley value may indicate the degree to which each alarm predicts or is correlated with incidence of a particular type of anomaly.

[0057] It should be noted that score unit 160 may produce score data other than Shapley scores in some embodiments. For example, score unit 160 may produce score data using a linear regressor and / or a probability density function. Other examples of scores include feature importance scores and scores computed using attribution methods. In another example, score data may be determined as a mathematical correlation between the alarm data (or genealogy data) and the rejection data. In yet other examples, score data may be determined as a feature importance using information gain, as a feature importance using number of times a feature is used in iterative modelling, and / or as importance scores from LIME modelling.

[0058] FIGs. 3A-3B are flow charts showing two exemplary computerized methods for monitoring a manufacturing system for medical devices, according to some embodiments. Computerized methods 300 and 320 may be performed by a computer system having at least one processor and computer memory storing processor-executable instructions that, when executed by the at least one processor, cause the at least one processor to perform the steps discussed below. Referring first to FIG. 3A, method 300 begins at step 302, where a computer system receives alarm data representing incidence of alarms resulting from irregular operation of manufacturing machinery. The alarm data may be received from a plurality of sensors (e.g., sensors 110). Each alarm belongs to one of a plurality of alarmtypes, and each alarm type is associated with a different type of irregular operation of the manufacturing machinery.

[0059] At step 304, the computer system receives information indicative of whether a medical device (e.g., a syringe filled with medicine) of a plurality of batches of medical devices is to be rejected as being compromised due to a type of anomaly detected by one or more cameras (e.g., camera(s) 121).

[0060] At step 306, the computer system generates system effectiveness data for each batch of medical devices calculated based at least in part on a rate at which medical devices in the batch were rejected (either due to a specific type of anomaly, or due to any anomaly).

[0061] Optionally, at step 308, the computer system selects a subset of the batches of medical devices using the system effectiveness data. These batches may be classified as anomalous by the computer system, or the batches may be classified as anomalous based on manual user input (e.g., a user selection).

[0062] At step 310, the computer system generates a function ( / ) that fits the system effectiveness data relative to the alarm data. As such, function / describes the relationship between rejection data produced by rejection unit 125 and alarm data produced by sensors 110.

[0063] At step 312, the computer system generates score data using function / The score data comprises, for each alarm type of the plurality of alarm types, a score representing the extent to which incidence of alarms of the alarm type is a root cause responsible for the generated system effectiveness data. In embodiments that include step 308, the generated score data represent the extent to which incidence of alarms of the alarm type is a root cause responsible for the generated system effectiveness data within the selected subset of batches.

[0064] Referring now to FIG. 3B, computerized method 320 begins at step 322, where a computer system receives information indicative of whether a medical device (e.g., a syringe filled with medicine) of a plurality of batches of medical devices output by manufacturing machinery is to be rejected as being compromised due to a type of anomaly detected by one or more cameras (e.g., camera(s) 121).

[0065] At step 324, the computer system generates system effectiveness data for each batch of medical devices indicative of a rate at which medical devices in the batch were rejected (either due to a specific type of anomaly or due to any anomaly).

[0066] Optionally, at step 326, the computer system selects a subset of the batches of medical devices using the system effectiveness data. These batches may be classified as anomalous by the computer system.

[0067] At step 328, the computer system receives genealogy data representing genealogy information associated with the batches of medical devices.

[0068] At step 330, the computer system generates a function (g) that fits the system effectiveness data relative to the genealogy data. As such, function g describes the relationship between system effectiveness data produced by rejection unit 125 and genealogy data produced by genealogy data unit 150.

[0069] At step 332, the computer system generates score data using function g. The score data represent an extent to which the genealogy information associated with the batches of medical devices is a root cause responsible for the generated system effectiveness data.

[0070] As discussed in connection with FIG. 1, classification unit 130 may select a subset of the batches inspected by inspection machinery 120 that are classified as anomalous. This operation may be performed using one or more types of classifiers. One example is binary classifier 132. Another example is severity ranking classifier 134. Other types of classifiers may be used. FIG. 4 is a graphical user interface (GUI) showing various panels configured to allow users to select subsets of batches of medical devices. GUI 200 includes a binary classifier panel 212, which allows a user to select batches using binary classifier 132. GUI 200 further includes a severity ranking classifier panel 214, which allows a user to select batches using severity ranking classifier 134. It should be noted that not all GUIs 200 need to include both panels. For example, some GUIs 200 may only include panel 212 and other GUIs 200 may only include panel 214.

[0071] Each panel includes a plot, where batch identifier axis 206 serves as the x-axis and system effectiveness axis 208 serves as the y-axis. The panels plot the system effectiveness data received by classification unit 130 against batches of medical devices. In other words, for each batch identifier displayed in the x-axis, there is a corresponding system effectiveness score. In one embodiment, for instance, the system effectiveness score may simply equal the rejection rate due to a specific type of anomaly. Anomaly type selector 210 allows the user to select one among multiple types of anomalies to use in calculating the system effectiveness score, examples of which are provided above. In this example, the user has selected the anomaly type associated with missing needles. It should be noted that the plots illustrated in panels 212 and 214 change as the user selects different types of anomalies. This is becausethe rejection rate associated with a particular type of anomaly is generally different from the rejection rate associated with another type of anomaly.

[0072] User selection menu 220 allows the user to vary the batches to be displayed in the panels. Selecting the button “Inclusion - Range selector’’ allows the user to display batches with identifiers that are within a particular range. Selecting the button “Exclusion - Range selector” allows the user to exclude batches with identifiers that are within a particular range. Selecting the button “Include - Individual batches” allows the user to include batches with specified identifiers. Selecting the button “Exclude - Individual batches” allows the user to exclude batches with specified identifiers.

[0073] Referring first to binary classifier panel 212, in some embodiments, batch selection may be performed on the basis of a threshold value. In the displayed embodiment, the system effectiveness axis is configured to show rejection rates associated with the type of anomaly selected in anomaly selector 210. In this example, batches associated with rejection rates exceeding the threshold value may be selected and classified as anomalous, while batches associated with rejection rates below the threshold value may not be selected and may be classified as non-anomalous. In some embodiments, the system effectiveness axis may instead be configured to show z-scores. A z-score is a statistical measurement that describes a value's relationship to the mean of a group of values. It is measured in terms of standard deviations from the mean. The z-score indicates how many standard deviations an element is from the mean. A z-score may be calculated in accordance with the following formula:

[0074] In the formula, X represents the rejection rate being evaluated, p represents the mean rejection rate of a population of batches and o represents the standard deviation of the population. The z-scores may be selected over a sliding window comprising system effectiveness data for a fixed number of batches. In the sliding window, batch identifiers are changed over time, though the number of displayed batches remains constant. In some embodiments, the z-scores may be compared to a threshold; batches associated with rejection rates having z-scores exceeding the threshold value may be selected and classified as anomalous, while batches associated with rejection rates having z-scores below the threshold value may not be selected and may be classified as non-anomalous.

[0075] Referring now to severity ranking classifier panel 214, a severity ranking classifier may classify batches based on the degrees of severity and may select batches based on the classification. For example, as shown in FIG. 4, a severity ranking classifier may classifybatches in accordance with four states (although other number of states are possible). The states are ranked from the most severe to the least severe, and batches are placed into the four buckets depending on the severity of the rejected rate. In some embodiments, only the batches that have been placed in the bucket representing the highest severity (or in the two buckets representing the highest severity, or in the three buckets representing the highest severity, etc.) are ultimately selected. In some embodiments, selecting the subset of the batches of medical devices comprises (1) classifying the plurality of batches of medical devices in two or more classes characterized by different levels of severity, (2) displaying the plurality of batches of medical devices with identifiers identifying the respective classification, (3) receiving input from a user intended to select one or more among the displayed plurality of batches of medical devices; and (4) selecting the subset of the batches of medical devices based on the input received from the user. In some embodiments, a Hidden Markov Model (HMM) may be used to rank batches according to levels of severity. The model may be trained using historical batches. For example, each new batch produced by the manufacturing system may be added to the training data, resulting in the HMM model being continually updated based on historical rejection rates. The HMM model may be retrained regularly, for example once per week.

[0076] FIG. 5 displays identifiers of batches that have been selected using classification unit 130 in accordance with one example. These batches (identified with numeral 500 in FIG. 5) are classified as anomalous, and the regressions performed in the following steps may be limited to these batches.

[0077] FIGs. 6A-6C are exemplary GUIs shown on a display device to visualize score data.The GUI of FIG. 6A shows a table including score data associated with the selected subset 500. In this case, scores are performed between alarm types and anomaly types, for example using computerized method 300. As in FIG. 4, an anomaly type selector (610) allows a user to select a type of anomaly among several possible options. In the example of FIG. 6 A, the user has selected the anomaly type associated with missing needles. The scores shown in the table represent the extent to which occurrence of the various types of alarms is a root cause responsible for the selected anomaly type. As discussed above in connection with FIG. 1 , each type of alarm is associated with a different type of irregular operation of the manufacturing machinery. The table of FIG. 6 A lists the following examples of alarm types: “guard door inadvertently open,” “insufficient force on plunger,” “spilled liquid” and “inaccurate position of syringe.” Other types of alarms may be listed in addition to or in lieuof the types of alarms listed in the table. Each score represents the extent to which one of the listed alarm types is a root cause responsible for the selected anomaly type - the larger the score, the higher the degrees of causality. Scores may be determined using score unit 160, examples of which are described above.

[0078] FIG. 6B is another graphical user interface showing a table including score data associated with the selected subset 500. In this case, scores are performed between genealogy data and anomaly types, for example using computerized method 320. As in FIG. 6A, anomaly type detector 610 allows a user to select a type of anomaly among several possible options. In the example of FIG. 6B, the user has selected the anomaly type associated with the presence of foreign particles. The scores shown in the table represent of the extent to which genealogy associated with the selected batch is a root cause responsible for the selected anomaly type. The table of FIG. 6B lists the following examples of genealogy data: “vendor,” “material,” and “country manufactured.” Other types of genealogy may be listed in additional to or in lieu of those listed in the table. Scores may be determined using genealogy regressor 144, examples of which are described above.

[0079] FIG. 6C is another graphical user interface showing a table including score data associated with the selected subset 500. FIG. 6C combines elements of FIG. 6A with elements of FIG. 6B. More specifically, the table of FIG. 6C shows scores between the selected anomaly type and various types of alarms as well as scores between the selected anomaly type and various types of genealogy associated with the selected batch. In this way, the user may assess different types of scores in the same table. The column labeled “feature” identifies alarm types and genealogy types. The column labeled “feature type” identifies whether the associated feature is of an alarm or a genealogy.

[0080] Scores determined using the techniques described herein may provide an indication as to which types of irregularities in the manufacturing machinery have a greater impact on the rate at which medical devices are ultimately rejected, and therefore, on the generated system effectiveness data. This information is insightful because it enables operators to perform targeted maintenance on the manufacturing line. Addressing irregularities that produce high correlations has a greater chance to reduce the rejection rate than addressing irregularities that produce low correlations. This information may also be helpful in improving training and / or evaluation of manufacturing personnel or teams of manufacturing personnel. For example, if certain alarms are determined to have a high impact on the rate at which medical devices are rejected (and therefore, on the generated system effectivenessdata), then the average time required to address or resolve occurrences of such alarms may be determined and compared across multiple teams (e.g., shifts) of manufacturing personnel. For example, if a certain type of alarm X is determined to have a high impact on overall system effectiveness, lots of the start and stop times of alarm X may be analyzed across multiple teams (e.g., shifts) of manufacturing personnel to determine an average time required to resolve incidences of alarm X. Through this analysis, teams that have both a higher-than- average as well as a lower-than-average response time may be identified. This, in turn, enables best practices and insights to be shared between teams to bring under-performing teams up to the same standards as over-performing teams. The observed distribution of time required to resolve incidences of alarm X may also be compared across multiple teams (e.g., shifts) of manufacturing personnel, and analyzed using known statistical methods, to determine whether variances in average time required to resolve such incidences are statistically significant or not.

[0081] Additionally or alternatively, scores determined using the techniques described herein may provide an indication as to the origin of batches of medical devices having a greater impact on the rate at which medical devices are ultimately rejected, and therefore, on the generated system effectiveness data. This information is insightful because it enables operators to assess whether a particular group of medical devices should be discarded a priori.

[0082] In some embodiments, the computer system may use score data to automatically determine whether to reject the selected subset of batches. The determination is automatic in that the computer system makes the determination without input from the user. The automatic determination may be performed, for example, in accordance with a set of predefined rules.

[0083] FIG. 7 shows an illustrative implementation of a computer system that may be used to perform any of the aspects of the techniques and embodiments disclosed herein, including computerized methods 300 and 320. An illustrative implementation of a computer system 700 that may be used to perform any of the aspects of the techniques and embodiments disclosed herein is shown in FIG. 7. The computer system 700 may be configured to perform various methods and acts as described in FIGs. 3A-3B, and may be used to implement the units shown in FIG. 1 (rejection unit 125, classification unit 130, regression unit 140, genealogy data unit 150 and score unit 160). The computer system 700 may include on or more processors 710, and one or more non-transitory computer-readable storage media (e.g., memory 720 and one or more non-volatile storage media 730) and a display 740. Theprocessor 710 may control writing data to and reading data from the memory 720 and the non-volatile storage device 730 in any suitable manner, as the aspects of the disclosure described herein are not limited in this respect. In some embodiments, the computer system 700 may also be a complete system on module (SOM), such as NVIDIA’s Jetson module, which include CPU, GPU, memory, and any other components in a system. In some embodiments, the computer system 700 may be located at any suitable site. For example, the computer system 700 may be co-located with the manufacturing system 100 or may be on the network. In other variations, the system may not need to include a memory, but instead programming instructions are running on one or more virtual machines or one or more containers on a cloud. For example, the various methods illustrated above may be implemented by a server on a cloud that includes multiple virtual machines, each virtual machine having an operating system, a virtual disk, virtual network and applications, and the programming instructions for detecting defects in manufactured components may be stored on one or more of those virtual machines on the cloud.

[0084] To perform functionality and / or techniques described herein, the processor 710 may execute one or more instructions stored in one or more computer-readable storage media (e.g. the memory 720, storage media, etc.), which may serve as non-transitory computer-readable storage media storing instructions for execution by the processor 710. Any other software, programs or instructions described herein may be stored and executed by computer system 700. It will be appreciated that computer code may be applied to any aspects of methods and techniques described herein.

[0085] The various methods or processes outlined herein may be coded as software that is executable on one or more processors that employ any one of a variety of operating systems or platforms. Additionally, such software may be written using any of numerous suitable programming languages and / or programming or scripting tools, and also may be compiled as executable machine language code or intermediate code that is executed on a virtual machine or a suitable framework.

[0086] In this respect, various inventive concepts may be embodied as at least one non- transitory computer readable storage medium (e.g., a computer memory, one or more floppy discs, compact discs, optical discs, magnetic tapes, flash memories, circuit configurations in Field Programmable Gate Arrays or other semiconductor devices, etc.) encoded with one or more programs that, when executed on one or more computers or other processors, implement the various embodiments of the present disclosure. The non-transitory computer-readable medium or media may be transportable, such that the program or programs stored thereon may be loaded onto any computer resource to implement various aspects of the present disclosure as discussed above.

[0087] The terms “program,” “software,” and / or “application” are used herein in a generic sense to refer to any type of computer code or set of computer-executable instructions that can be employed to program a computer or other processor to implement various aspects of embodiments as discussed above. Additionally, it should be appreciated that according to one aspect, one or more computer programs that when executed perform methods of the present disclosure need not reside on a single computer or processor, but may be distributed in a modular fashion among different computers or processors to implement various aspects of the present disclosure.

[0088] Computer-executable instructions may be in many forms, such as program modules, executed by one or more computers or other devices. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Typically, the functionality of the program modules may be combined or distributed as desired in various embodiments.

[0089] Also, data structures may be stored in non-transitory computer-readable storage media in any suitable form. Data structures may have fields that are related through location in the data structure. Such relationships may likewise be achieved by assigning storage for the fields with locations in a non-transitory computer-readable medium that convey relationship between the fields. However, any suitable mechanism may be used to establish relationships among information in fields of a data structure, including through the use of pointers, tags or other mechanisms that establish relationships among data elements.

[0090] It is to be understood that the disclosed subject matter is not limited in its application to the details of construction and to the arrangements of the components set forth in the description or illustrated in the drawings. The disclosed subject matter is capable of other embodiments and of being practiced and carried out in various ways. Also, it is to be understood that the phraseology and terminology employed herein are for the purpose of description and should not be regarded as limiting.

[0091] As such, those skilled in the art will appreciate that the conception, upon which this disclosure is based, may readily be utilized as a basis for the designing of other structures, methods, and systems for carrying out the several purposes of the disclosed subject matter. It is important, therefore, that the description provided herein be regarded as including suchequivalent constructions insofar as they do not depart from the spirit and scope of the disclosed subject matter.

[0092] Use of ordinal terms such as “first,” “second,” “third.” etc., in the claims to modify a claim element does not by itself connote any priority, precedence, or order of one claim element over another or the temporal order in which acts of a method are performed, but are used merely as labels to distinguish one claim element having a certain name from another element having a same name (but for use of the ordinal term) to distinguish the claim elements.

[0093] Also, the phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting. The use of “including,” “comprising,” “having,” “containing,” “involving,” and variations thereof herein, is meant to encompass the items listed thereafter and equivalents thereof as well as additional items.

[0094] The word “exemplary” is used herein to mean serving as an example, instance, or illustration. Any embodiment, implementation, process, feature, etc. described herein as exemplary should therefore be understood to be an illustrative example and should not be understood to be a preferred or advantageous example unless otherwise indicated.

[0095] Although the disclosed subject matter has been described and illustrated in the foregoing exemplary embodiments, it is understood that the present disclosure has been made only by way of example, and that numerous changes in the details of implementation of the disclosed subject matter may be made without departing from the spirit and scope of the disclosed subject matter.

[0096] Various aspects are described in this disclosure, which include, but are not limited to, the following aspects:

[0097] (1) A manufacturing system, comprising manufacturing machinery configured to output a plurality of batches of medical devices; a plurality of sensors configured to generate alarm data representing incidence of alarms resulting from irregular operation of the manufacturing machinery during manufacturing of each batch of medical devices, wherein each alarm belongs to one of a plurality of alarm types, and each alarm type is associated with a different type of irregular operation of the manufacturing machinery; one or more cameras configured to visually inspect each medical device output by said manufacturing machinery and determine whether to reject said medical device as being compromised due to a detected type of anomaly out of one or more possible types of anomalies; and at least one computer processor and computer memory storing processor-executable instructions that,when executed by the at least one computer processor, cause the at least one computer processor to: generate system effectiveness data for each batch of medical devices output by the manufacturing machinery, wherein the system effectiveness data is calculated based at least in part on a rate at which medical devices in said batch were rejected, generate a function that fits the system effectiveness data relative to the alarm data, and generate score data using the function, the score data comprising, for each alarm type of the plurality of alarm types, an associated score representing an extent to which incidence of alarms of said alarm type is a root cause responsible for the system effectiveness data.

[0098] (2) The manufacturing system of aspect 1, wherein the manufacturing machinery comprises equipment for dispensing medicine, and wherein outputting the batches of medical devices comprises outputting a plurality of medical devices filled with the medicine using the equipment from a plurality of empty medical devices.

[0099] (3) The manufacturing system of aspect 1, wherein the instructions, when executed by the at least one computer processor, are further configured to cause the at least one computer processor to select a subset of the batches of medical devices, and wherein each score in the generated score data represent an extent to which incidence of alarms of the alarm type associated with said score is a root cause responsible for the generated system effectiveness data within said selected subset of batches.

[0100] (4) The manufacturing system of aspect 3, wherein selecting the subset of the batches of medical devices comprises: performing a comparison between the system effectiveness data and a threshold value; and selecting the subset of the batches of medical devices based on the comparison.

[0101] (5) The manufacturing system of aspect 3, wherein selecting the subset of the batches of medical devices comprises: determining a z-score based on the system effectiveness data for each of at least some of the plurality of batches of medical devices; and selecting the subset of the batches of medical devices based on the z-scores.

[0102] (6) The manufacturing system of aspect 5, wherein determining the z-score for each of at least some of the plurality of batches of medical devices is performed over a sliding window comprising a fixed number of batches of medical devices.

[0103] (7) The manufacturing system of aspect 5, wherein the instructions, when executed by the at least one computer processor, are further configured to cause the at least one computer processor to compare the z-scores to a threshold, and wherein selecting the subset of the batches of medical devices is performed based on the comparison.

[0104] (8) The manufacturing system of aspect 3, wherein selecting the subset of the batches of medical devices comprises: classifying the plurality of batches of medical devices in two or more classes characterized by different levels of severity based on the system effectiveness data; and selecting the subset of the batches of medical devices based on the classification.

[0105] (9) The manufacturing system of aspect 8, wherein classifying the plurality of batches of medical devices in classes characterized by different levels of severity comprises executing a Hidden Markov Model (HMM) using the system effectiveness data, wherein the HMM is trained with past system effectiveness data associated with the manufacturing system.

[0106] (10) The manufacturing system of aspect 3, wherein selecting the subset of the batches of medical devices comprises: classifying the plurality of batches of medical devices in two or more classes characterized by different levels of severity based on the system effectiveness data; displaying the plurality of batches of medical devices with identifiers identifying the respective classification; receiving input from a user intended to select one or more among the displayed plurality of batches of medical devices; and selecting the subset of the batches of medical devices based on the input received from the user.

[0107] (11) The manufacturing system of aspect 3, wherein the stored processor-executable instructions, when executed by the at least one computer processor, further cause the at least one computer processor to: control a display device to visualize the score data in conjunction with the selected subset of the batches of medical devices, and / or use the score data to determine whether to reject the selected subset of the batches of medical devices.

[0108] (12) The manufacturing system of aspect 1, wherein the one or more possible types of anomalies include a plurality of types of anomalies, and the one or more cameras is configured to determine whether to reject said medical device as being compromised due to any one or more of the plurality of types of anomalies.

[0109] (13) The manufacturing system of aspect 12, wherein the plurality of types of anomalies comprises at least one selected from the group consisting of: a missing or damaged needle in a medical device; a missing or damaged plunger in the medical device; a damaged glass sidewall in the medical device; and presence of particles in the medical device.

[0110] (14) The manufacturing system of aspect 1, wherein the plurality of alarm types includes at least one of: insufficient force to push a syringe; spilling or dripping of liquid; and improper dimension in a medical device.

[0111] (15) The manufacturing system of aspect 1 , wherein generating the score data comprises computing Shapley Additive Explanations.

[0112] (16) The manufacturing system of aspect 1, wherein the system effectiveness data is calculated based on a rate at which medical devices in said batch of medical devices were rejected to detection of one type of anomaly out of the one or more possible types of anomalies.

[0113] (17) The manufacturing system of claim 1, wherein the system effectiveness data is calculated based on a rate at which medical devices in said batch of medical devices were rejected due to detection of any type of anomaly out of the one or more possible types of anomalies.

[0114] (18) A manufacturing system, comprising: manufacturing machinery configured to output a plurality of batches of medical devices; one or more cameras configured to visually inspect each medical device output by said manufacturing machinery and determine whether to reject said medical device as being compromised due to a detected type of anomaly out of one or more possible types of anomalies; and at least one computer processor and computer memory storing processor-executable instructions that, when executed by the at least one computer processor, cause the at least one computer processor to: generate system effectiveness data for each batch of medical devices output by the manufacturing machinery, wherein the system effectiveness data is calculated based at least in part on a rate at which medical devices in said batch were rejected, receive genealogy data representing genealogy information associated with the batches of medical devices, generate a function that fits the system effectiveness data relative to the genealogy data, and generate score data using the function, the score data comprising a score representing an extent to which the genealogy information associated with the batches of medical devices is a root cause responsible for the system effectiveness data.

[0115] (19) The manufacturing system of aspect 18, wherein the manufacturing machinery comprises equipment for dispensing medicine, and wherein outputting the batches of medical devices comprises outputting a plurality of medical devices filled with the medicine using the equipment from a plurality of empty medical devices.

[0116] (20) The manufacturing system of aspect 18, wherein the instructions, when executed by the at least one computer processor, are further configured to cause the at least one computer processor to select a subset of the batches of medical devices, and wherein the generated score data represent an extent to which the genealogy information associated withthe batches of medical devices is a root cause responsible for the generated system effectiveness data within said selected subset of batches.

[0117] (21) The manufacturing system of aspect 20, wherein selecting the subset of the batches of medical devices comprises: performing a comparison between the system effectiveness data and a threshold value; and selecting the subset of the batches of medical devices based on the comparison.

[0118] (22) The manufacturing system of aspect 20, wherein selecting the subset of the batches of medical devices comprises: determining a z-score based on the system effectiveness data for each of at least some of the plurality of batches of medical devices; and selecting the subset of the batches of medical devices based on the z-scores.

[0119] (23) The manufacturing system of aspect 20, wherein determining the z-score for each of at least some of the plurality of batches of medical devices is performed over a sliding window comprising a fixed number of batches of medical devices.

[0120] (24) The manufacturing system of aspect 20, wherein selecting the subset of the batches of medical devices comprises: classifying the plurality of batches of medical devices in two or more classes characterized by different levels of severity based on the system effectiveness data; and selecting the subset of the batches of medical devices based on the classification.

[0121] (25) The manufacturing system of aspect 24, wherein classifying the plurality of batches of medical devices in classes characterized by different levels of severity comprises executing a Hidden Markov Model (HMM) using the system effectiveness data, wherein the HMM is trained with past system effectiveness data associated with the manufacturing system.

[0122] (26) The manufacturing system of aspect 20, wherein selecting the subset of the batches of medical devices comprises: classifying the plurality of batches of medical devices in two or more classes characterized by different levels of severity based on the system effectiveness data; displaying the plurality of batches of medical devices with identifiers identifying the respective classification; receiving input from a user intended to select one or more among the displayed plurality of batches of medical devices; and selecting the subset of the batches of medical devices based on the input received from the user.

[0123] (27) The manufacturing system of aspect 20, wherein the stored processorexecutable instructions, when executed by the at least one computer processor, further cause the at least one computer processor to: control a display device to visualize the score data inconjunction with the selected subset of the batches of medical devices, and / or use the score data to determine whether to reject the selected subset of the batches of medical devices.

[0124] (28) The manufacturing system of aspect 16, wherein the one or more possible types of anomalies include a plurality of types of anomalies, and the one or more cameras is configured to determine whether to reject said medical device as being compromised due to any one or more of the plurality of types of anomalies.

[0125] (29) The manufacturing system of aspect 28, wherein the plurality of types of anomalies comprises at least one selected from the group consisting of: a missing or damaged needle in a medical device; a missing or damaged plunger in the medical device; a damaged glass sidewall in the medical device; and presence of particles in the medical device.

[0126] (30) A method for monitoring a manufacturing system, comprising: receiving, from a plurality of sensors, alarm data representing incidence of alarms resulting from irregular operation of manufacturing machinery, wherein the manufacturing machinery is configured to output a plurality of batches of medical devices, wherein each alarm belongs to one of a plurality of alarm types, and each alarm type is associated with a different type of irregular operation of the manufacturing machinery; receiving infomration indicative of whether a medical device of the plurality of batches of medical devices is to be rejected as being compromised due to a type of anomaly out of one or more possible types of anomalies detected by one or more cameras; generating system effectiveness data for each batch of medical devices output by the manufacturing machinery, wherein the system effectiveness data is calculated based at least in part on a rate at which medical devices in said batch were rejected; generating a function that fits the system effectiveness data relative to the alarm data; and generating score data using the function, the score data comprising, for each alarm type of the plurality of alarm types, an associated score representing an extent to which incidence of alarms of said alarm type is a root cause responsible for the system effectiveness data.

[0127] (31) The method of aspect 30, wherein the manufacturing machinery comprises equipment for dispensing medicine, and wherein the manufacturing machinery is configured to output a plurality of batches of medical devices by: outputting a plurality of medical devices filled with the medicine using the equipment from a plurality of empty medical devices.

[0128] (32) The method of aspect 30, further comprising selecting a subset of the batches of medical devices, and wherein each score in the generated score data represent an extent towhich incidence of alarms of the alarm type associated with said score is a root cause responsible for the generated system effectiveness data within said selected subset of batches.

[0129] (33) The method of aspect 32, wherein selecting the subset of the batches of medical devices comprises: performing a comparison between the system effectiveness data and a threshold value; and selecting the subset of the batches of medical devices based on the comparison.

[0130] (34) The method of aspect 32, wherein selecting the subset of the batches of medical devices comprises: determining a z-score based on the system effectiveness data for each of at least some of the plurality of batches of medical devices; and selecting the subset of the batches of medical devices based on the z-scores.

[0131] (35) The method of aspect 34, wherein determining the z-score for each of at least some of the plurality of batches of medical devices is performed over a sliding window comprising a fixed number of batches of medical devices.

[0132] (36) The manufacturing system of aspect 34, further comprising comparing the z- scores to a threshold, and wherein selecting the subset of the batches of medical devices is performed based on the comparison.

[0133] (37) The method of aspect 32, wherein selecting the subset of the batches of medical devices comprises: classifying the plurality of batches of medical devices in two or more classes characterized by different levels of severity based on the system effectiveness data; and selecting the subset of the batches of medical devices based on the classification.

[0134] (38) The method of aspect 37, wherein classifying the plurality of batches of medical devices in classes characterized by different levels of severity comprises executing a Hidden Markov Model (HMM) using the system effectiveness data, wherein the HMM is trained with past system effectiveness data associated with the manufacturing system.

[0135] (39) The method of aspect 32, wherein selecting the subset of the batches of medical devices comprises: classifying the plurality of batches of medical devices in two or more classes characterized by different levels of severity based on the system effectiveness data; displaying the plurality of batches of medical devices with identifiers identifying the respective classification; receiving input from a user intended to select one or more among the displayed plurality of batches of medical devices; and selecting the subset of the batches of medical devices based on the input received from the user.

[0136] (40) The method of aspect 32, further comprising: controlling a display device to visualize the score data in conjunction with the selected subset of the batches of medicaldevices, and / or using the score data to determine whether to reject the selected subset of the batches of medical devices.

[0137] (41) The method of aspect 30, wherein the one or more possible types of anomalies include a plurality of types of anomalies, and the one or more cameras is configured to determine whether to reject said medical device as being compromised due to any one or more of the plurality of types of anomalies.

[0138] (42) The method of aspect 41, wherein the plurality of types of anomalies comprises at least one selected from the group consisting of: a missing or damaged needle in a medical device: a missing or damaged plunger in the medical device; a damaged glass sidewall in the medical device; and presence of particles in the medical device.

[0139] (43) The method of aspect 30, wherein the plurality of alarm types include at least one of: insufficient force to push a syringe; spilling or dripping of liquid; and improper dimension in a medical device.

[0140] (44) The method of aspect 30, wherein generating the score data comprises computing Shapley Additive Explanations.

[0141] (45) A method for monitoring a manufacturing system, comprising: receiving information indicative of whether a medical device of a plurality of batches of medical devices output by manufacturing machinery is to be rejected as being compromised due to a type of anomaly out of one or more possible types of anomalies detected by one or more cameras; generating system effectiveness data for each batch of medical devices output by the manufacturing machinery, wherein the system effectiveness data is calculated based at least in part on a rate at which medical devices in said batch were rejected; receiving genealogy data representing genealogy information associated with the batches of medical devices; generating a function that fits the system effectiveness data relative to the genealogy data, and generating score data using the function, the score data comprising a score representing an extent to which the genealogy information associated with the batches of medical devices is a root cause responsible for the detected type of anomaly.

[0142] (46) The method of aspect 45, wherein the manufacturing machinery comprises equipment for dispensing medicine, and wherein the manufacturing machinery is configured to output a plurality of batches of medical devices by: outputting a plurality of medical devices filled with the medicine using the equipment from a plurality of empty medical devices.

[0143] (47) The method of aspect 45, further comprising selecting a subset of the batches of medical devices, and wherein the generated score data represent an extent to which the genealogy information associated with the batches of medical devices is a root cause responsible for the generated system effectiveness data within said selected subset of batches.

[0144] (48) The method of aspect 47, wherein selecting the subset of the batches of medical devices comprises: performing a comparison between the system effectiveness data and a threshold value; and selecting the subset of the batches of medical devices based on the comparison.

[0145] (49) The method of aspect 47, wherein selecting the subset of the batches of medical devices comprises: determining a z-score based on the system effectiveness data for each of at least some of the plurality of batches of medical devices; and selecting the subset of the batches of medical devices based on the z-scores.

[0146] (50) The method of aspect 49, wherein determining the z-score for each of at least some of the plurality of batches of medical devices is performed over a sliding window comprising a fixed number of batches of medical devices.

[0147] (51) The method of aspect 47, wherein selecting the subset of the batches of medical devices comprises: classifying the plurality of batches of medical devices in two or more classes characterized by different levels of severity based on the system effectiveness data; and selecting the subset of the batches of medical devices based on the classification.

[0148] (52) The method of aspect 51, wherein classifying the plurality of batches of medical devices in classes characterized by different levels of severity comprises executing a Hidden Markov Model (HMM) using the system effectiveness data, wherein the HMM is trained with past system effectiveness data associated with the manufacturing system.

[0149] (53) The method of aspect 47, wherein selecting the subset of the batches of medical devices comprises: classifying the plurality of batches of medical devices in two or more classes characterized by different levels of severity based on the system effectiveness data; displaying the plurality of batches of medical devices with identifiers identifying the respective classification; receiving input from a user intended to select one or more among the displayed plurality of batches of medical devices; and selecting the subset of the batches of medical devices based on the input received from the user.

[0150] (54) The method of aspect 47, further comprising controlling a display device to visualize the score data in conjunction with the selected subset of the batches of medicaldevices, and / or using the score data to determine whether to reject the selected subset of the batches of medical devices.

[0151] (55) The method of aspect 45, wherein the one or more possible types of anomalies include a plurality of types of anomalies, and the one or more cameras is configured to determine whether to reject said medical device as being compromised due to any one or more of the plurality of types of anomalies.

[0152] (56) The method of aspect 55, wherein the plurality of types of anomalies comprises at least one selected from the group consisting of: a missing or damaged needle in a medical device: a missing or damaged plunger in the medical device; a damaged glass sidewall in the medical device; and presence of particles in the medical device.

[0153] (57) The method of aspect 45, wherein generating the score data comprises computing Shapley Additive Explanations.

Claims

CLAIMSWhat is claimed is:

1. A manufacturing system, comprising:manufacturing machinery configured to output a plurality of batches of medical devices;a plurality of sensors configured to generate alarm data representing incidence of alarms resulting from irregular operation of the manufacturing machinery during manufacturing of each batch of medical devices, wherein each alarm belongs to one of a plurality of alarm types, and each alarm type is associated with a different type of irregular operation of the manufacturing machinery;one or more cameras configured to visually inspect each medical device output by said manufacturing machinery and determine whether to reject said medical device as being compromised due to a detected type of anomaly out of one or more possible types of anomalies; andat least one computer processor and computer memory storing processor-executable instructions that, when executed by the at least one computer processor, cause the at least one computer processor to:generate system effectiveness data for each batch of medical devices output by the manufacturing machinery, wherein the system effectiveness data is calculated based at least in part on a rate at which medical devices in said batch were rejected,generate a function that fits the system effectiveness data relative to the alarm data, andgenerate score data using the function, the score data comprising, for each alarm type of the plurality of alarm types, an associated score representing an extent to which incidence of alarms of said alarm type is a root cause responsible for the system effectiveness data.

2. The manufacturing system of claim 1, wherein the instructions, when executed by the at least one computer processor, are further configured to cause the at least one computer processor to select a subset of the batches of medical devices, and wherein each score in the generated score data represent an extent to which incidence of alarms of the alarm type associated with said score is a root cause responsible for the generated system effectiveness data within said selected subset of batches.

3. The manufacturing system of claim 2, wherein selecting the subset of the batches of medical devices comprises:performing a comparison between the system effectiveness data and a threshold value; and selecting the subset of the batches of medical devices based on the comparison.

4. The manufacturing system of claim 2, wherein selecting the subset of the batches of medical devices comprises:determining a z-score based on the system effectiveness data for each of at least some of the plurality of batches of medical devices; andselecting the subset of the batches of medical devices based on the z-scores.

5. The manufacturing system of claim 2, wherein selecting the subset of the batches of medical devices comprises:classifying the plurality of batches of medical devices in two or more classes characterized by different levels of severity based on the system effectiveness data; and selecting the subset of the batches of medical devices based on the classification.

6. The manufacturing system of claim 5, wherein classifying the plurality of batches of medical devices in classes characterized by different levels of severity comprises executing a Hidden Markov Model (HMM) using the system effectiveness data, wherein the HMM is trained with past system effectiveness data associated with the manufacturing system.

7. The manufacturing system of claim 1, wherein the stored processor-executable instructions, when executed by the at least one computer processor, further cause the at least one computer processor to:control a display device to visualize the score data.

8. The manufacturing system of claim 1, wherein the one or more possible types of anomalies includes at least one of:a missing or damaged needle in a medical device;a missing or damaged plunger in the medical device:a damaged glass sidewall in the medical device; andpresence of particles in the medical device.

9. The manufacturing system of claim 1, wherein the plurality of alarm types includes at least one of:insufficient force to push a syringe;spilling or dripping of liquid; andimproper dimension in a medical device.

10. The manufacturing system of claim 1, wherein generating the score data comprises computing Shapley Additive Explanations.

11. The manufacturing system of claim 1, wherein the system effectiveness data is calculated based on a rate at which medical devices in said batch of medical devices were rejected to detection of one type of anomaly out of the one or more possible types of anomalies.

12. The manufacturing system of claim 1, wherein the system effectiveness data is calculated based on a rate at which medical devices in said batch of medical devices were rejected due to detection of any type of anomaly out of the one or more possible types of anomalies.

13. A manufacturing system, comprising :manufacturing machinery configured to output a plurality of batches of medical devices;one or more cameras configured to visually inspect each medical device output by said manufacturing machinery and detennine whether to reject said medical device as being compromised due to a detected type of anomaly out of one or more possible types of anomalies; andat least one computer processor and computer memory storing processor-executable instructions that, when executed by the at least one computer processor, cause the at least one computer processor to:generate system effectiveness data for each batch of medical devices output by the manufacturing machinery, wherein the system effectiveness data is calculated based at least in part on a rate at which medical devices in said batch were rejected,receive genealogy data representing genealogy information associated with the batches of medical devices,generate a function that fits the system effectiveness data relative to the genealogy data, andgenerate score data using the function, the score data comprising a score representing an extent to which the genealogy information associated with the batches of medical devices is a root cause responsible for the system effectiveness data.

14. The manufacturing system of claim 13, wherein the instructions, when executed by the at least one computer processor, are further configured to cause the at least one computer processor to select a subset of the batches of medical devices, and wherein the generated score data represent an extent to which the genealogy information associated with the batches of medical devices is a root cause responsible for the generated system effectiveness data within said selected subset of batches.

15. The manufacturing system of claim 14, wherein selecting the subset of the batches of medical devices comprises:performing a comparison between the system effectiveness data and a threshold value; and selecting the subset of the batches of medical devices based on the comparison.

16. The manufacturing system of claim 14, wherein selecting the subset of the batches of medical devices comprises:determining a z-score based on the system effectiveness data for each of at least some of the plurality of batches of medical devices; andselecting the subset of the batches of medical devices based on the z-scores.

17. The manufacturing system of claim 14, wherein selecting the subset of the batches of medical devices comprises:classifying the plurality of batches of medical devices in two or more classes characterized by different levels of severity based on the system effectiveness data; andselecting the subset of the batches of medical devices based on the classification.

18. The manufacturing system of claim 17, wherein classifying the plurality of batches of medical devices in classes characterized by different levels of severity comprises executing a Hidden Markov Model (HMM) using the system effectiveness data, wherein the HMM is trained with past system effectiveness data associated with the manufacturing system.

19. The manufacturing system of claim 13, wherein the stored processor-executable instructions, when executed by the at least one computer processor, further cause the at least one computer processor to:control a display device to visualize the score data.

20. The manufacturing system of claim 13, wherein the one or more possible types of anomalies includes at least one of:a missing or damaged needle in a medical device;a missing or damaged plunger in the medical device;a damaged glass sidewall in the medical device; andpresence of particles in the medical device.

Images