Medical device defect detection and prediction system

Abstract

Technology is disclosed for detecting and predicting defects in devices using image analysis and machine learning. In one implementation, a system generates a testing sample comprising a video segment representing a device undergoing inspection. The video segment is processed to determine testing sample features, which are utilized to determine a likelihood of a defect in the device. The system employs defect detection logic, which may include a machine learning model trained on defect signatures, to analyze the testing sample features. Based on the analysis, the system determines if a defect is likely present and can provide a notification or segregate a potentially defective device. The technology also includes methods for generating device profiles with defect detection logic corresponding to particular types of devices and instructions for reproducing controlled testing conditions, enabling adaptive and improved defect detection capabilities across various types of devices.

Description

MEDICAL DEVICE DEFECT DETECTION AND PREDICTION SYSTEM BACKGROUND OF THE INVENTION

[0001] Manufactured devices play critical roles across various industries, from healthcare and aerospace to consumer electronics and industrial machinery. However, these devices can be susceptible to defects arising from various sources, including manufacturing errors, use of substandard materials, improper packaging, or mishandling prior to deployment. Such defects, often subtle and difficult to detect initially, may manifest as critical failures later when the device is in use, potentially leading to severe consequences.

[0002] The importance of thorough device inspection cannot be overstated, especially for applications where failure can result in significant harm or financial loss. For instance, in medical settings, device failures can jeopardize patient safety, while in extreme environments like space or deep-sea operations, malfunctions can lead to catastrophic outcomes. However, many devices, once deployed, become inaccessible for routine inspection or replacement. This is particularly true for implanted medical devices and devices used in extreme environments such as components installed on space crafts, complex machinery, or deep-sea equipment.SUMMARY OF THE INVENTION

[0003] This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used in isolation as an aid in determining the scope of the claimed subject matter.

[0004] Embodiments of the present disclosure are generally directed towards technologies for detecting defects in devices, such as medical devices. In particular, the technology disclosed herein facilitates device inspection and provides improved defect detection capability, which can include, in various implementations, detecting a defect in a device or determining a likelihood of a defect in the device, predicting a likelihood of a future failure of the device, or otherwise determining a device-related defect including a defective component, equipment, implant, instrument, or tool. Upon detection of a defect, or likely defect, a notification may be generated and automatically provided. In some implementations, the device determined to be defective (or likely defective) is automatically segregated from other non-defective devices or devices that have not undergone testing, or is automatically designated for segregation. Further, the defective device may be disposed of automatically,subjected to further testing such as to confirm the defect or obtain additional information about a defect, designated for manual inspection, or designated for repair, in various implementations.

[0005] According to one example embodiment and at a high level, a device defect detection system is provided and includes computer system operable for controlling testing conditions of a testing environment to generate a testing sample of a particular device. The testing environment includes an imaging system for obtaining images of the device undergoing testing. In some implementations, the imaging system is used to generate a video segment of the device or an aspect of the device, and includes a plurality of video frames. The testing sample, comprising the video segment representing the device, is generated according to the testing conditions. The testing sample may be processed to determine one or more data features of the testing sample, referred to herein as testing sample features. These testing sample features are utilized to determine a likelihood of a defect in the particular tested device. In some implementations, a testing sample feature represents an aspect of the device across multiple frames of the video segment, and may comprise a spatio-temporal data feature. Further, in some of these implementations and as further described herein, testing sample features that capture device-image information across the multiple frames are programmatically extracted or determined from the video segment using one or more models or computer-implemented processes.

[0006] Based on the testing sample features, a likelihood of a defect associated with the device is determined. In some embodiments, a defect or likelihood thereof is determined using a defect detection logic, for instance, by applying the defect detection logic to the testing sample features. In particular and as further described in various embodiments herein, defect detection logic comprises computer instructions for determining a defect or likelihood thereof, and may include one or more machine learning models or algorithms for the defect determination. In some embodiments, the defect detection logic is specific to a particular device type or version that is being tested for a defect. In some instances, the defect detection logic includes defect signatures or uses a machine learning model trained using defect signatures. A defect signature comprises one or more data features associated with a particular defect, and may include the most distinguishing data features for the particular defect. In some instances, the defect detection logic comprises computer instructions for determining a similarity between a testing sample feature and a defect signature, which can be used to determine a likelihood of a defect in the tested device.

[0007] In some implementations, a device profile is accessed and utilized by a computing system of the device detection system to facilitate testing a particular device type, according to the processes described herein, to determine a likelihood of a defect. A device profile may correspond to the particular device or device type being tested. Moreover, some embodiments of a device profile include defect detection logic corresponding to the particular type of device that is undergoing testing.

[0008] After determining that a tested device likely has a defect, a notification regarding the defect may be automatically generated and issued, as described herein. For instance, the notification is provided via a user interface to a clinician or testing operator. Additionally, the tested device may be automatically segregated for disposal, inspection, or for further testing, as described herein. In some embodiments, where a first tested device is determined as likely to have a particular defect, defect detection logic is programmatically updated, as described herein, to be more sensitive towards detecting the particular defect in the next tested device that is the same device type as the first tested device. In this way, these embodiments of the defect detection system are programmatically adaptive and enabled to learn to better detect particular defects based on prior detections.

[0009] Further, in this way, the various embodiments described herein improve capability of defect detection systems. In particular, the disclosed technology provides a solution to the limitations of conventional inspection methods by leveraging advanced videobased analysis and machine learning techniques. For example, by utilizing spatio-temporal features extracted from video segments, the system can detect subtle defects that may be missed by conventional static image analysis or human inspection.

[0010] Accordingly, in one aspect, the present disclosure provides a system for detecting or predicting defects in a medical device. The system comprises an imaging system operable to capture image data, a device manipulator operable to control a motion of the medical device, at least one computer processor, and computer memory having computer-readable instructions embodied thereon, that, when executed by the at least one computer processor, perform operations. The operations comprise receiving a device profile corresponding to the medical device, the device profile including instructions for controlling testing conditions for generating a testing sample corresponding to the medical device. The operations further comprise, based on the device profile, programmatically controlling the device manipulator to cause a first motion of the medical device. The operations further comprise causing the imaging system to generate the testing sample comprising a videosegment representing at least a portion of the medical device, the video segment generated by capturing image data of the medical device during the first motion, the video segment comprising a plurality of frames. The operations further comprise processing the video segment to determine, using the plurality of frames, at least one testing sample feature representing, across the plurality of frames, an aspect of the medical device, thereby forming a testing sample feature set. The operations further comprise, based on the testing sample feature set, determining a likelihood of a defect associated with the medical device. The operations further comprise based on the determined likelihood, causing a notification, regarding the defect associated with the medical device, to be provided via the system-user interface.

[0011] In another aspect, the disclosure provides a computing system for detecting or predicting defects in a device. The computing system comprises at least one computer processor, and computer memory having computer-readable instructions embodied thereon, that, when executed by the at least one computer processor, perform operations. The operations comprise accessing a video segment representing at least a portion of the device and comprising a plurality of frames. The operations further comprise processing the video segment to determine, using the plurality of frames, at least one testing sample feature representing, across the plurality of frames, an aspect of the device, thereby forming a testing sample feature set. The operations further comprise determining a defect associated with the device by applying a defect detection logic to the testing sample feature set, the defect detection logic corresponding to the device. The operations further comprise causing an indication to be provided regarding the defect associated with the device.

[0012] In another aspect, the disclosure provides a computer-implemented method for detecting or predicting defects in a device. The method comprises generating, using at least one camera, video data comprising a plurality of video segments, each video segment having an associated label and comprising a plurality of frames, the plurality of video segments including at least a first segment generated by capturing video of a defective instance of the device having a particular defect and having associated therewith a first segment label indicating the particular defect, and a second segment generated by capturing video of a nondefective instance of the device and having a second segment label indicating no defect, the first and second videos of the segments captured under a controlled condition including a first controlled motion of the defective and non-defective instances of the device with respect to the camera, or of the camera with respect to the defective and non-defective instances of the device. The method further comprises training a defect detection model to detect a defect featurecorresponding to the particular defect by using the plurality of video segments and their associated labels. The method further comprises generating, from the trained defect detection model, defect detection logic for detecting the particular defect associated with the device. The method further comprises generating machine-readable instructions regarding the controlled condition used for capturing the video and specifying the first controlled motion. The method further comprises generating a device profile corresponding to the device by associating an indication of the device with the defect detection logic and the machine-readable instructions. The method further comprises providing the device profile to be accessible for testing a candidate instance of the device for the particular defect.

[0013] The foregoing general description of the illustrative embodiments and the following detailed description thereof are merely exemplary aspects of the teachings of this disclosure and are not restrictive.BRIEF DESCRIPTION OF THE DRAWING

[0014] Aspects of the disclosure are described in detail below with reference to the attached drawing figures, wherein:

[0015] FIG. 1 depicts an example process flow for programmatically detecting a defect in a medical device, in accordance with an embodiment of the disclosure;

[0016] FIG. 2 is a diagram depicting an example computing architecture suitable for implementing aspects of the present disclosure;

[0017] FIG. 3A illustratively depicts aspects of one example of a medical device that is a stent, in accordance with an embodiment of the disclosure;

[0018] FIG. 3B illustratively depicts aspects of example defects on a device, in accordance with an embodiment of the disclosure;

[0019] FIG. 3C illustratively depicts aspects of a video segment representing a defective instance of a medical device, in accordance with an embodiment of the disclosure;

[0020] FIGS. 4-6 each depict a flow diagram of a method for detecting or predicting defects in a medical device, in accordance with an embodiment of the present disclosure;

[0021] FIG. 7 is a block diagram of an example computing environment suitable for use in implementing an embodiment of the present disclosure; and

[0022] FIG. 8 is a block diagram of an example computing environment suitable for use in implementing an embodiment of the present disclosure.DETAILED DESCRIPTION OF THE INVENTION

[0023] The subject matter of aspects of the present disclosure is described with specificity herein to meet statutory requirements. However, the description itself is not intended to limit the scope of this patent. Rather, it is contemplated that the claimed subject matter might also be embodied in other ways, to include different steps or combinations of steps similar to the ones described in this document, in conjunction with other present or future technologies. Moreover, although the terms “step” and / or “block” may be used herein to connote different elements of methods employed, the terms should not be interpreted as implying any particular order among or between various steps herein disclosed unless and except when the order of individual steps is explicitly described. Each method described herein may comprise a computing process that may be performed using any combination of hardware, firmware, and / or software. For instance, various functions may be carried out by a processor executing instructions stored in memory. The methods may also be embodied as computer-usable instructions stored on computer storage media. The methods may be provided by a stand-alone application, a service or hosted service (stand-alone or in combination with another hosted service), or a plug-in to another product, to name a few.

[0024] Various embodiments described herein are generally directed towards technologies for detecting defects in devices, such as medical devices. In particular, the technology disclosed herein facilitates device inspection and provides improved defect detection capability, which can include detecting a defect in a device or determining a likelihood of a defect in the device, predicting a likelihood of a future failure of the device, or otherwise determining a device-related defect including a defective component, equipment, implant, instrument, or tool. Examples of device defects include, without limitation, cracks or breaks; discoloration; burrs or spurs; weak areas; contaminants or foreign bodies; porosity; delamination; blistering or bubbles; corrosion; misalignment; pitting, dents, scratches, or surface flaws; warpage; distortion; or other malformations.

[0025] Upon detection of a defect, or likely defect, a notification may be generated and provided automatically. For example, a notification can be provided to a clinician, a caregiver, hospital administrator, or a manufacturer or supplier of the device. In some implementations, the notification comprises a report that includes information about the defect (or a likely defect), the particular device, and in some further instances, manufacturing information about the device, such as a serial number or identification number, an indication of a manufacturinglocation, a lot number, or a run number. In some implementations, the device determined to be defective (or likely defective) is automatically segregated from other non-defective devices or devices that have not undergone testing, or is automatically designated for segregation. Further, the defective device may be disposed of automatically, subjected to further testing such as to confirm the defect or obtain additional information about a defect, designated for manual inspection, or designated for repair, in various implementations.

[0026] In some embodiments, where a first tested device is determined as likely having a particular defect, the defect detection system is programmatically updated to be more sensitive towards detecting the particular defect in the next tested device that is the same device type as the first tested device. For example, the defect detection logic is updated based on learnings from a detected defect, as described herein. In particular, in some instances where a particular device is determined to have a particular type of defect, other devices of the same type may have a greater likelihood of having that same (or similar) type of defect. For example, if a faulty piece of equipment used in manufacturing the device causes a particular defect in a first manufactured device, such as a burr, it is possible that other devices manufactured using the same equipment will have the burr defect. Accordingly, these embodiments of the defect detection system are programmatically adaptive and enabled to learn to better detect particular defects based on prior detections.Overview of Technical Problems, Technical Solutions, and Technological Improvements

[0027] As described previously, manufactured devices can play critical roles across various industries, from healthcare and aerospace to consumer electronics and industrial machinery. However, these devices are susceptible to defects arising from various sources, including manufacturing errors, use of substandard materials, improper packaging, or mishandling prior to deployment. Moreover, the advancement of manufacturing technologies has led to the production of increasingly sophisticated devices across various industries. However, this progress has been accompanied by a persistent challenge: the detection of subtle defects that can compromise device integrity and safety. Despite rigorous quality control measures, manufactured devices remain susceptible to these defects. These flaws, often imperceptible during initial inspections, may manifest as critical failures when the device is in use, potentially leading to severe consequences.

[0028] The increasing complexity of devices and the trend towards miniaturization further complicate the inspection process. Conventional inspection approaches, including manual inspections and Automated Vision Inspection (AVI) systems, lack the sensitivity or otherwise often fall short in identifying all potential defects, especially those that only become apparent under specific conditions or over time. This limitation is particularly critical in industries such as medical device manufacturing, where the consequences of device failure can be life-threatening.

[0029] Another limitation of the conventional technology is that defect testing of devices is complex, time-consuming, and often requires identifying multiple defects at various locations on the device. This complexity often leads to testing only a random sample of devices for potential defects, which may not be sufficient to ensure the quality and safety of all manufactured devices.

[0030] Another limitation is the heavy reliance on manual inspection by humans, particularly in the manufacturing of complex devices such as medical devices. This approach is not only time-intensive but also prone to human error and inconsistency.

[0031] Another limitation of the conventional technology is that defect detection typically analyzes features of single images or isolated video frames to identify defects. This approach may miss subtle defects that arc only apparent when examining the device's characteristics across multiple consecutive frames.

[0032] Another limitation of conventional technology is that it often relies on capturing still image data of the device, even when the device is moved to capture various angles. This method fails to leverage the valuable information that can be gleaned from analyzing light reflection changes across consecutive video frames, which can be induced by controlled movement of the device, camera, or light source.

[0033] Yet another limitation of the conventional technology is that defect detection approaches are generally static and do not adapt based on prior detections. This lack of learning capability limits the system's ability to improve its accuracy and efficiency over time, especially in detecting emerging or evolving defect patterns.

[0034] In contrast to the above technical limitations, embodiments of the technology provided herein improve the capabilities of defect detection systems by leveraging particular video-based analysis and machine learning technologies. This approach enables a more comprehensive and accurate inspection process, allowing for the testing of every device or a significantly larger sample size, thereby enhancing overall quality control and safety assurance.Moreover, the need to rely on manual human inspection is reduced. This not only increases the speed and consistency of defect detection but also minimizes the potential for human error, especially in the inspection of complex devices such as medical equipment.

[0035] Another improvement provided by some embodiments of the defect detection technology utilize spatio-temporal features extracted from video segments, rather than relying solely on single images or isolated frames. By analyzing features across consecutive video frames, the system can detect subtle defects that may be missed by conventional static image analysis, providing a more robust assessment of device integrity.

[0036] Still another improvement provided by some embodiments of the defect detection technology incorporate controlled light reflection changes during a video capture process. This is achieved through precise manipulation of the device, camera, or light source, generating valuable information that is only available through the capture and analysis of consecutive video frames. This dynamic approach reveals defects that may not be apparent in static imaging conditions.

[0037] Still another improvement provided by some embodiments of the defect detection technology employs adaptive learning mechanisms. By continuously updating and refining the defect detection logic based on prior detections, the system becomes increasingly accurate and efficient over time, particularly in identifying emerging or evolving defect patterns.

[0038] Still another improvement provided by some embodiments of the defect detection technology provides a more comprehensive analysis by examining multiple aspects of the device simultaneously. This comprehensive approach allows for the detection of a multitude of defects at different locations on the device in a single inspection process, improving both the efficiency and thoroughness of quality control measures.Additional Description of the Embodiments

[0039] Turning now to FIG. 1, an example process flow 100 is provided showing aspects of a defect detection system for programmatically detecting a defect in a medical device, according to an embodiment of this disclosure. It should be understood that this and other arrangements described herein are set forth only as examples. Other arrangements and elements (for example, machines, interfaces, functions, orders, and groupings of functions) can be used in addition to or instead of those shown, and some elements can be omitted altogetherfor the sake of clarity. Further, many of the elements described herein are functional entities that are implemented as discrete or distributed components or in conjunction with other components, and in any suitable combination and location. Various functions described herein as being performed by one or more entities are carried out by hardware, firmware, and / or software. For instance, some functions are carried out by a processor executing instructions stored in memory.

[0040] According to the example embodiment of process flow 100 and at a high level, a device defect detection system is provided and includes a computing system 111 operable for controlling aspects of a testing environment 160 to generate a testing sample of a particular device 121. Computing system 111 comprises one or more computing devices, such as computing device 700 described in FIG. 7, and may be embodied as a client computing device and / or server, or a distributed computing system, such as distributed computing system 800 described in FIG. 8. For instance, computing system 111 may be provided via multiple devices arranged in a distributed environment that collectively provide the functionality described herein. Additionally, other components not shown may also be included within the distributed environment.

[0041] The example testing environment 160 includes an imaging system 135 for obtaining images of the device undergoing testing. Imaging system 135 includes one or more cameras configured to generate a video segment of the tested device 121, or an aspect of tested device 121, such as a component or portion of the device. The video segment includes a plurality of video frames, in some instances, such as example video segment 350 described in connection with FIG. 3C.

[0042] Example process flow 100 also illustratively depicts a set of devices 120 comprising medical devices to be tested for defect. In this example, devices 120 include instances of a particular type of stent used to open and support a narrowed or blocked blood vessel or duct within the body of a patient. Tested device 121 represents one instance of the devices 120. It should be understood that, although the example embodiment of process flow 100 is described with reference to a medical device, and specifically stents, it is contemplated that embodiments of the technologies described herein are applicable to many other types of devices, and are particularly advantageous for devices that are mass produced and difficult to inspect, repair, or replace once they are put into use. For example and without limitation, these include patient implants, internal machinery components particularly difficult to inspect orservice, such as ball bearings, and components to be used in extreme environments such as in deep-sea equipment or spacecraft systems.

[0043] In some implementations, aspects of the imaging system 135 are controlled by computing system 111 to generate the testing sample of a particular device that is tested, such as tested device 121. For example, various properties of the cameras are controlled such as a frame rate, shutter speed, spectrum range, depth of field, distance of a camera lens to the device, and the number of cameras used. Some embodiments of imaging system 135 or computer system 111 include an imaging system controller, such as imaging system controller 230 described in connection with FIG. 2.

[0044] In some implementations, the testing sample is generated by capturing image data of the tested device 121 under specific testing conditions of the testing environment 160. One such example type of a testing condition includes manipulating the tested device 121 to cause a motion or movement of the tested device 121 as the image data is being captured. For instance, a movement of the tested device 121 may be a rotational motion, a linear motion, a planar motion, a tilt motion, or a combination of one or more of these or other motions. The device may be manipulated (i.e., caused to move) via a device manipulator 155. For example, the device manipulator 155 may comprise a robotic arm, a conveyer belt, a roller, a vibrating plane, a disc, or other means for causing a controlled motion of one or more of the devices to undergo defect inspection. Some embodiments of computer system 111 include a device manipulator controller, such as device manipulator controller 250 described in connection with FIG. 2.

[0045] Another aspect of testing environment 160 that is controlled in some embodiments, is the lighting, which illuminates the tested device 121 as image data is being captured. In these embodiments, testing environment 160 further includes a lighting system 145 comprising one or more light sources that illuminate the tested device 121. Some implementations of lighting system 145 or computer system 111 further include a lighting controller, such as lighting controller 240 described in connection with FIG. 2, for controlling various properties of the lighting in the testing environment 160. Examples of lighting properties that may be controlled include, without limitation, the number of light sources illuminating the tested device 121, an angle of incidence of a particular light source with regard to the tested device 121, lighting color temperature, lighting strobe frequency, and / or lumen level, wavelength, or independent orientation of the light separate from the device.

[0046] Continuing with example process flow 100, using imaging system 135 and computing system 111, a testing sample, comprising a video segment representing the tested device 121, is generated according to the testing conditions controlled in testing environment 160. The testing sample is processed by computing system 111 to determine one or more data features of the testing sample, referred to herein as testing sample features. The testing sample features are utilized to determine a likelihood of a defect in the particular tested device 121. In some embodiments, a testing sample feature includes a data feature, such as a feature vector, determined from the testing sample. Alternatively or in addition, a testing sample feature includes at least a portion of the video segment of a testing sample.

[0047] In some implementations, a testing sample feature represents an aspect of the tested device 121 across multiple frames of the video segment, and may comprise a spatiotemporal data feature. For example, in some embodiments, a data feature corresponding to a region of interest (ROI) on an image of the tested device 121 is identified, for example in a first frame. The data feature is also detected in the ROI across at least one additional frame. In some instances and in regards to each frame, the data feature comprises a spatial feature. For example, aspects of spatial features may include, without limitation, information regarding pixel edges, intensity, textures, color, shape features, interest points, and spatial distribution. In some implementations, an ROI is detected in the frames using pixel segmentation or aspects thereof, such as edge detection or thresholding. Further, in some of these implementations and as further described herein, testing sample features that capture device-image information across the multiple frames are programmatically extracted or determined from the video segment using one or more models or computer-implemented processes operating on computing system 111.

[0048] For instance, in one example implementation, an optical flow technique is employed by computing system 111 to compute a motion of a feature or an object between consecutive frames, capturing temporal information about movement. These motion vectors can then be used alone, or as additional features alongside a spatial feature extracted from individual frames, thus providing a spatio-temporal feature. Alternatively, or in addition, a higher-order motion model or higher-order trajectory model is utilized. For instance, polynomial regression can be fitted to the motion data across multiple frames. In this way, a smoother approximation of the trajectory of an object, such as a defect-related image feature, is provided. Moreover, the resulting motion trajectory can be more robust to noise and provide a clearer representation of the object over time. This enhanced clarity facilitates detection evenwhen the object is not consistently visible in every frame. The trajectory vectors or smooth motion vectors can then be used alone, or as additional features alongside spatial features extracted from individual frames to provide a spatio-temporal feature. In this way, by combining spatial features with smooth motion vectors or trajectory vectors, a more comprehensive representation is provided of an object’s behavior or characteristics in the video segment, which can improve defect detection, as well as defect-type classification. In some instances, the object that is tracked between frames is the tested device 121 or an aspect of the tested device 121, such as an image artifact detected from a particular frame of the video segment.

[0049] In another example implementation, an object tracking model is employed by computing system 111 to determine a testing sample feature from a video segment representing the tested device 121. In some instances, the object tracking model is utilized by determining an image artifact or a region of interest regarding an aspect of the tested device 121 that is depicted in a first frame of the video segment, and tracking the artifact or region of interest across multiple frames. In one embodiment, a ByteTrack object tracking model is utilized to determine testing sample features. Alternatively other implementations with object tracking utilize Simple Online and Realtime Tracking (SORT), Deep Simple Online and Realtime Tracking (DccpSORT), or similar object tracking models.

[0050] It still other example implementations, a three-dimensional Convolutional Neural Network (3D CNN), a recurrent neural network (RNN), and / or a video transformer are employed by computer system 111 to determine a testing sample feature from the video segment. For implementations employing an RNN, the RNN may comprise a Convolutional Long Short-Term Memory (ConvLSTM) or an Inflated 3D ConvNet (I3D). For some implementations, each of these approaches comprises a model that is trained to determine spatio-temporal features associated with a particular type of defect. In some instances, the models are trained under a supervised learning approach using labeled video data representing a device having the particular type of defect. The labeled video data is generated by capturing image data of the defective device under controlled conditions similar to the testing conditions of testing environment 160 for detecting the defect in the tested device 121. These similar testing conditions may include similar lighting, image capturing, and motion, for example.

[0051] Continuing with the example process flow 100, the testing sample features determined by computing system 111 are then utilized to determine a likelihood of a defect associated with the tested device 121. In some implementations, a defect or likelihood thereofis determined using a defect detection logic 118, for instance, by employing computing system 111 to apply the defect detection logic 118 to the testing sample features. In some implementations, this includes applying the defect detection logic 118 to the testing sample video segment(s) or portion thereof. In other embodiments, the defect detection logic 118 is applied to one or more data features, such as a feature vector, determined from processing the testing sample video segment(s) by computer system 111.

[0052] In particular, defect detection logic 118 comprises computer instructions for determining a defect or likelihood thereof, and may include one or more machine learning models or algorithms for the defect determination. For example, a particular machine learning model of the defect detection logic 118 may be trained on data derived from instances of the specific device type that includes defective and non-defective instances of the devices 120. For instance, the training data may comprise video data representing defective and non-defective instances of the devices 120, wherein the video data is captured under conditions similar to the testing conditions of testing environment 160 used to generate the testing sample video segment(s). In some embodiments, the defect detection logic 118 is specific to a particular device type or version of tested device 121 that is being tested for a defect. For example, in some embodiments, defect detection logic 118 includes computer instructions for identifying a region of interest on tested device 121 and may further include instructions for detecting a data feature that is associated with the region of interest and is indicative of a possible defect.

[0053] In some embodiments, the defect detection logic 118 includes defect signatures orrises a machine learning model trained using defect signatures. A defect signature comprises one or more data features associated with a particular defect, and may include the most distinguishing data features for the particular defect. In some embodiments, a defect signature includes video data representing one or more instances of a particular defect occurring in a defective version of a device 120. In some instances, the video data is captured under conditions similar to the testing conditions of testing environment 160 used to generate the testing sample video segment(s).

[0054] For example, a particular machine learning model that is trained on the data derived from defective and non-defective instances of the device 120 can be utilized to detect particular data features associated with a particular defect. As further described herein, examples of these models include, without limitation, 3D CNNs, RNNs, which may comprise a ConvLSTM or I3D, and video transformers. In some embodiments, two of these models are employed in a Siamese Neural Network architecture. For example, some embodiments of thedefect detection logic 118 include a Siamese Neural Network comprising two 3D CNNs, two ConvLSTMs, or two-stream networks that share the same weights. In other words, both models will detect the same types of features from input video segment data; the models (or one model that is copied) may be trained to detect the features using labeled training data derived from defective and non-defective instances of the device 120. In particular, the Siamese Neural Network is trained using a contrastive loss function or triplet loss, using the labeled training data, in order to distinguish similarity and differences in the training data. Once trained the Siamese Neural Network may be deployed as defect detection logic 118. For example, in one implementation, the testing sample data feature and a defect signature are input into the Siamese Neural Network, which determines a measure of similarity between the two inputs, wherein the measure of similarity corresponds to the likelihood that the particular type of defect associated with the defect signature is present in the tested device 121.

[0055] In some embodiments, a defect signature is determined using feature extraction from a model that is trained on the data derived from defective and non-defective instances of the device 120. Moreover, according to some embodiments, particular distinguishing features to include in a defect signature are determined based on a feature importance score, if available in the model, or using a suitable feature extraction or feature determination techniques, such as attention mechanisms, activation mapping or feature mapping, autocncodcrs for feature extraction, SHapley Additive exPlanations (SHAP), Local Interpretable Model-Agnostic Explanations (LIME), or any suitable technique for extracting or otherwise determining distinguishing data features for the particular defect, which may include feature vectors that are spatio-temporal.

[0056] In some embodiments, a defect signature is determined by capturing video data of defective instances of the device 120 that has a particular defect. The captured video data is processed by a computer system, such as computer system 110, into one or more video segments, each segment representing the particular defect. Accordingly, there can exist multiple defect signatures for a particular type of device, such that each defect signature corresponds to a different type or variation of defect that can occur in the particular type of device. The video data used for determining the defect signature may be captured under conditions similar to the testing conditions of testing environment 160 used to generate the testing sample video segment(s).

[0057] Alternatively or in addition, a defect signature may be manually determined or may be derived from historic data regarding the particular device type, including instances offailures, or may be determined from records of testing sample features of a device 120, such as described herein, where a device defect is detected or a defect later occurs.

[0058] In some instances, the defect detection logic 118 comprises computer instructions, executable by computing system 111, for determining a similarity between a testing sample feature and a defect signature. This determination of similarity is used to determine a likelihood of a defect in the tested device 121. For instance, a higher similarity corresponds to a greater likelihood of the defect associated with the defect signature being present in the tested device 121. In some embodiments, the similarity is determined based on a binary-match decision. In other embodiments, similarity is determined based on a feature similarity distance between the testing sample feature set and the defect signature. The feature similarity distance may be computed by computing system 111 using Euclidean Distance, Cosine Similarity, Dynamic Time Warping, Hamming Distance, or other techniques for the determining of feature similarity distance.

[0059] In some embodiments, the computed similarity is applied to a threshold by computing system 111. Based on satisfying or exceeding the threshold, the tested device is determined to likely have a defect (see item 180 in FIG. 1 indicating a predicted defect). In some instances, the threshold is predetermined. In some instances, the threshold is based on the particular device type or the particular defect. In one embodiment, the threshold corresponds to a probability of 0.6, and in another embodiment the threshold is 0.5+, representing more likely than not. Accordingly, by way of example and without limitation, for a predetermined similarity threshold used for testing a particular type of device for a particular type of defect, where a computed similarity between a testing sample corresponding to the particular type of device and a defect signature associated with the particular type of defect exceeds the similarity threshold, it is determined that the tested device 121 for which the testing sample was derived likely has the particular type of defect. Moreover, the likelihood of the device having the defect corresponds to the degree of similarity, in some implementations.

[0060] In some implementations, a device profile is accessed and utilized by computing system 111 to facilitate testing a particular device type, according to the processes described herein, to determine a likelihood of a defect. A device profile may correspond to the particular tested device 121 or device type being tested. In some embodiments, a device profile includes defect detection logic 118 corresponding to the particular type of device that is undergoing testing. In some embodiments, a device profile may be accessed by the defect detection testing system via a quick reference (QR) code on the tested device 121 or its packaging. For example,the imaging system 135 used to generate a video segment also may be used to scan the QR code on (or associated with) the tested device 121.

[0061] In various embodiments, a device profile includes one or more of the following: computer instructions, executable by computing system 111 , for detecting a defect that occurs in a device type corresponding to the device profile; computer instructions for controlling testing conditions of testing environment 160, such as recreating lighting, motion, imagecapture configuration, or other conditions in which training video data is generated for defect detection models or defect signatures; computer instructions for generating and providing a notification based on a likelihood of a defect being determined in the tested device 121, which may include content to include in the notification, an indication of a recipient for receiving the notification, and / or formatting instructions; instructions or resources for disposing or repairing a tested device 121 that is determined to likely be defective; and instructions for additional testing to be performed based on a determined likelihood of a defect.

[0062] In particular and in regards to performing additional testing, where an initial testing determines that a particular tested device 121 likely has a defect or that there is a sufficient possibility of a defect, some embodiments of a device profile include computer instructions for performing additional testing on the tested device 121 to confirm (or to increase confidence of) a determination that the tested device 121 likely has a defect. In some instances, a sufficient possibility of a defect may be inferred wherein a similarity distance is less than a defect detection threshold (e.g., a similarity threshold) but satisfies a second threshold that is lower, implying less similarity exists. For instance, in one embodiment, the additional instructions include generating a second video segment under similar testing conditions and repeating the process to determine a likely defect. In other words, double-checking the initial determination. In another embodiment, the additional instructions include generating an additional video segment under different testing conditions, such as controlling device manipulator 155 to manipulate the tested device 121 in a different motion than a motion used for the first video segment, as the additional video segment is generated, and using a defect detection logic 118 corresponding to the different testing conditions.

[0063] Where it is determined that a tested device 121 likely does not have a defect (see item 170 in FIG. 1 indicating an insignificant likelihood of defect is determined), the tested device 121 may be designated at block 175 as not defective, or more specifically, as no defect detected. For instance, where a similarity threshold is not satisfied, or where defect detection logic 118 does not detect defect features in the tested sample(s) of tested device 121, then testeddevice 121 is determined as likely not having the particular defect(s) for which it was tested. In some implementations, a record regarding the testing and / or outcome, which may include the testing conditions and testing date, is created and stored in association with a device identification (Device ID) corresponding to the particular tested device 121. In some implementations, the tested device 121 is cleared for use.

[0064] Where it is determined that a tested device likely has a defect (see item 180 in FIG. 1 indicating a predicted defect), then according to the example embodiment of FIG. 1, at block 185 the notification regarding the defect is automatically generated and issued. In some instances, the notification is provided via a user interface to a clinician or testing operator. For example, the notification is presented on a graphical user interface (GUI) of the defect detection system, such as operator / clinician user interface (UI) 220, described in connection with FIG. 2 or presentation component 716, described in connection with FIG. 7. In some implementations, the notification includes information regarding the defect and may further include a portion of the video segment of the tested device 121 showing the features likely indicating a defect. In some implementations, the tested device 121 is designated at block 185 as likely defective. For example, a record regarding the testing and / or outcome, which may include the testing conditions and testing date, is created and stored in association with a device identification (Device ID) corresponding to the tested device 121. The record may further include the information regarding the defect.

[0065] In some implementations, a tested device 121 determined to be defective (or likely defective) is automatically segregated from other non-defective devices or devices 120 that have not undergone testing, or is automatically designated for segregation. Further, the defective device may be disposed of automatically, subjected to further testing such as to confirm the defect or obtain additional information about a defect, designated for manual inspection, or designated for repair, in various implementations. For example, a device ID such as a serial number, or other device identification on the device, or associated with the device, is associated with an indication of the likely defect and an indication to segregate the device from other devices that may be intended for use, to repair the device, to manually inspect the device, to subject the device to further testing, or to dispose of the device. In some implementations, the device is automatically rejected or automatically sorted for disposal.

[0066] In some embodiments, following completion of the defect detection process for a particular tested device 121, at block 190, process flow 100 continues by testing the next device 120. The defect testing proceeds at block 190 in a manner similar to the testing processesdescribed above for evaluating tested device 121. In some embodiments, where a first tested device 121 is determined by computing system 111 to be likely to have a particular defect, defect detection logic 118 is programmatically updated (by computing system 111) to be more sensitive towards detecting the particular defect in the next tested device that is the same device type as the first tested device 121. In particular, in some instances where a particular device is determined to have a particular type of defect, other devices of the same type may have a greater likelihood of having that same (or similar) type of defect. For example, if a faulty piece of equipment used in manufacturing the device causes a particular defect in a first manufactured device, such as a burr, it is possible that other devices manufactured using the same equipment will have the burr defect. Accordingly, defect detection logic 118 is updated based on learnings from a detected defect. For example and without limitation, defect detection logic 118 is updated by adding an ROI at a location corresponding to a detected defect; for an existing ROI at the location corresponding to a detected defect, adding or adjusting a weighting so that data features from the ROI have increased significance; updating a coefficient corresponding to a defect signature (or data feature associated with a defect) of a defect detection model, or fine-tuning the model, such as by adding a layer associated with detecting data features of the particular defect; lowering or adjusting a similarity threshold of defect detection logic 118 used for determining similarity between a testing sample feature and a defect signature corresponding to the particular defect so that a determined similarity (and thus a defect detection) is more likely to occur. In other words, the defect detection is biased to detect the particular defect. In this way, these embodiments of the defect detection system are programmatically adaptive and enabled to learn to better detect particular defects based on prior detections.

[0067] Referring now to FIG. 2, with continuing reference to FIG. 1, a block diagram is provided showing aspects of an example computing system architecture suitable for implementing an embodiment of a defect detection system and designated generally as system 200. System 200 represents one example of a suitable computing system architecture for programmatically inspecting a device to determine a defect. Other arrangements and elements can be used in addition to or instead of those shown, and some elements may be omitted for the sake of clarity. Further, as with operating environment 100, many of the elements described herein are functional entities that may be implemented as discrete or distributed components or in conjunction with other components, and in any suitable combination and location. In some embodiments, computing system 111 described in connection with FIG. 1 is embodied usingthe computing architecture of system 200 or includes at least some of the components of system 200. In another example, the computing device 700 of FIG. 7 or the distributed computing system 800 of FIG. 8 are employed to perform aspects of the system 200.

[0068] Example system 200 includes network 110, which includes, without limitation, one or more local area networks (LANs); a data bus or other internal network architecture, such as in the case when two or more components of system 200 operate on a single computer device; and / or one or more wide area networks (WANs). In one example, network 110 comprises the internet, intranet, and / or a cellular network, amongst any of a variety of possible public and / or private networks. Network 110 communicatively couples components of system 200, including imaging system controller 230 communicatively coupled to imaging system 235, lighting system controller 240 communicatively coupled to lighting system 245, device manipulator controller 250 communicatively coupled to device manipulator 255, testing sample(s) processing 270, operator / clinician UI 220, defect detector 280, defect detection logic generator 260, and storage 205. Although imaging system 235, lighting system 245, and device manipulator 255 are shown as communicatively coupled to controllers 230, 240, and 250, respectively, it is contemplated that these, and other components of example system 200 are communicatively coupled to any of the other components of system 200, as needed, via network 110.

[0069] Some components of example system 200 including: imaging system controller 230, lighting system controller 240, device manipulator controller 250, testing sample(s) processing 270 (including subcomponents 272 and 274), operator / clinician UI 220, defect detector 280, and defect detection logic generator 260 are embodied, in some implementations, as compiled computer instructions or functions, program modules, computer software services, or an arrangement of processes carried out on one or more computer systems, such as computing device 700, described in connection with FIG. 7, which may comprise a client computing device and / or a server. In some embodiments, these components of system 200 are distributed across a network, including one or more servers and / or client devices in the cloud, such as described in connection with FIG. 8, or reside on a client device, such as an end-user computing device. Moreover, functions performed by these components of system 200 or services carried out by these components if system 200 can be implemented at appropriate abstraction layer(s) such as the operating system layer, application layer, hardware layer, etc., of the computing system(s). Alternatively, or in addition, the functionality of these components and / or the embodiments described herein can be performed, at least in part, by one or morehardware logic components. For example and without limitation, illustrative types of hardware logic components include Field-Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application-Specific Standard Products (ASSPs), System-on-a-Chip systems (SOCs), Programmable Logic Controllers (PLCs), Complex Programmable Logic Devices (CPLDs), and so forth. Additionally, although functionality is described herein with regard to specific components shown in example system 200, it is contemplated that in some embodiments, functionality of these components is shared or distributed across other components.

[0070] Continuing with FIG. 2, operator / clinician user interface (UI) 220 is generally responsible for facilitating operation by a user of a defect detection system, such as the example device defect detection systems described in connection with FIGS. 1 and 2. In various implementations, operator / clinician UI 220 is embodied as a presentation component, such as presentation component 716 described in FIG. 7, and / or an input / output (I / O) component, such as I / O component 720 described in FIG. 7. Some embodiments of operator / clinician UI 220 comprise a graphical user interface (GUI). Operator / clinician UI 220 may comprise one or more applications or services on a client computing device, such as computing device 700, described in connection with FIG. 7, across multiple computing devices, or in the cloud environment. For example, in one embodiment, operator / clinician UI 220 manages the provision of information, such as notifications and alerts, to a user, such as an operator or clinician.

[0071] In some embodiments, operator / clinician UI 220 generates user interface elements associated with or used to facilitate presenting aspects of other components of system 200, such as controllers 230, 240, and 250, defect notifications or testing results from defect detector 280, and information about tested devices from a device profile 210. Similarly, in some embodiments operator / clinician UI 220 includes user interface functionality to receive input from a user, such as a clinician or system operator, in connection with operating the defect detection system. In particular, some embodiments of operator / clinician UI 220 present user interface elements to facilitate receiving input. By way of example and without limitation, these inputs can include information regarding particular devices for defect testing, various configuration parameters used for testing devices such as particular test routines to perform, settings for testing environment 160, and parameters associated with imaging system 235, lighting system 245, and / or device manipulator 255; defect detection settings, such as similarity thresholds; queries regarding testing results; or any other user input regarding the operation of- l -the example defect detection system. Such user interface elements and functionality for presenting information and receiving input may include graphical or audio interface elements (such as icons or indicators, graphics buttons, sliders, menus, sound, audio prompts, alerts, alarms, vibrations, pop-up windows, notification bar or status bar items, in-app notifications, or other similar features for interfacing with a user), queries, and prompts. Some embodiments of operator / clinician UI 220 employ speech synthesis, text-to-speech, or similar functionality for generating and presenting speech to the user. Further, some embodiments of operator / clinician UI 220 utilize a conversational model or language model, such as a large language model (LLM), to facilitate communication with a user. For instance, in one embodiment, an LLM is employed to provide functionality enabling a user to interact with the defect detection system. In particular, the user can ask questions about likely defects that are determined, and in response, receive answers comprising explanations formulated by the LLM describing aspects of detected defects. Similarly, in some implementations using a language model, functionality is provided enabling a user, following a determination of a likely defect, to ask what additional tests should be performed to confirm the defect or determine additional information regarding the likely defect.

[0072] In some embodiments, aspects of operator / clinician UI 220 are implemented using imaging system 235. For example, one embodiment described herein uses a camera of imaging system 235 to capture device ID infomration, such as a QR code or serial number, for a device 121 to be tested. Operator / clinician UI 220 is also used, in some embodiments, to facilitate training a defect detection model, which may include generating labeled data such as defect signatures. For example, an embodiment of operator / clinician UI 220 includes functionality enabling a user to manually label (or confirm a programmatically determined labeling of) training data including video segments showing particular defects of a device and video segments showing no defect. Some embodiments of operator / clinician UI 220 also include functionality for system diagnostics, such as conducting testing on or monitoring aspects of the operation of the defect detection system to ensure that the defect detection system is operating properly, or for maintaining or upgrading aspects of the defect detection system.

[0073] Example system 200 also includes various controllers for controlling aspects of testing environment 160. These components include imaging system controller 230, lighting system controller 240, and device manipulator controller 250, which operate image capture, lighting conditions, and device motion, respectively, during an inspection process.

[0074] Some embodiments of these controllers operate to control image capture, lighting conditions, and / or device motion using computerized (or machine) instructions for controlling a testing environment in which a device is undergoing inspection. These computerized instractions may comprise signals or machine commands for operating imaging system 235, lighting system 245, or device manipulator 255, or may comprise computer-readable instructions for use by imaging system controller 230, lighting system controller 240, or device manipulator controller 250 to control the operation of imaging system 235, lighting system 245, or device manipulator 255, respectively. In some embodiments, such as example system 200, the computerized instractions comprise testing system controller logic 214, and may be included in a device profile 210 that is associated with a particular type of device that is undergoing defect inspection. For instance, in some embodiments, particular computerized instructions for controlling aspects of the image capture, which may include instructions regarding video segment generation; lighting; or motion of a device are predetermined based on the particular device type. In some instances, these computerized instractions are determined based on conditions present during a creation of defect signatures for particular defects or training data used for training a defect detection model. In this way the instructions are used during testing of a device to create testing conditions that mimic the conditions under which defect signatures arc generated or training data for defect detection models is generated.

[0075] Accordingly, some embodiments of imaging system controller 230, lighting system controller 240, and / or device manipulator controller 250 work in concert to create testing conditions that closely mimic those used to generate defect signatures or training data for a defect detection model. This similarity in conditions ensures that the testing samples produced are highly comparable to the training data, thereby enhancing the accuracy and reliability of the defect detection process. By precisely controlling these environmental factors, the system can recreate scenarios that are most likely to reveal subtle defects, improving the overall effectiveness of the inspection.

[0076] Imaging system controller 230 is generally responsible for controlling various aspects of imaging system 235 to generate testing samples of devices undergoing defect inspection. In some implementations, imaging system controller 230 manages properties of one or more cameras in imaging system 235, such as frame rate, shutter speed, spectrum range, depth of field, and distance of a camera lens to the device being tested. Imaging system controller 230 may also coordinate the timing and duration of image capture to align with the movements of the device under inspection or changes in lighting conditions.

[0077] Lighting system controller 240 is generally responsible for managing the lighting conditions during the defect inspection process. This controller may adjust various properties of lighting system 245, including but not limited to the number of light sources illuminating the tested device, the angle of incidence of particular light sources, lighting color temperature, lighting strobe frequency, and lumen level. By precisely controlling these lighting parameters, lighting system controller 240 enables the creation of optimal conditions for detecting subtle defects that may only be visible under specific lighting scenarios.

[0078] Device manipulator controller 250 is generally responsible for controlling the motion and positioning of one or more devices undergoing inspection via device manipulator 255. In some implementations, device manipulator controller 250 operates various types of motion, such as rotational, linear, tilt, and planar movements, or combinations thereof. In some embodiments, device manipulator controller 250 coordinates with imaging system controller 230 and / or lighting system controller 240 to control a device's movement in a manner that is synchronized with particular aspects image capture and lighting conditions. Such synchronization can facilitate generating video segments that capture the device from particular angles or conditions, which in some instances can be caused to change during generation of a video segment, thereby facilitating the detection of defects that may only be apparent during specific movements or from certain perspectives.

[0079] Imaging system 235 is operated to capture image data of devices undergoing defect inspection. Embodiments of imaging system 235 comprise one or more cameras configured to capture digital image information used to generate one or more video segments representing the tested device or aspects thereof. In some implementations, imaging system 235 comprises one or more digital image sensors, such as Charge-Coupled Device (CCD), Complementary Metal-Oxide-Semiconductor (CMOS), Foveon X3 sensors, or other similar sensors or combinations thereof for digitally capturing image information. As depicted in example system 200, imaging system 235 is controlled by imaging system controller 230, for instance, to control various properties or settings of the imaging system 235, such as the number of cameras, frame rate, shutter speed, spectrum range, depth of field, and / or camera positioning relative to the device under inspection.

[0080] Some embodiments of imaging system 235 and / or imaging system controller 230 use the image data generated by imaging system 235 and to generate one or more video segments representing the tested device or aspects thereof. Further, some of these embodiments utilize testing system controller logic 214 to generate the video segments. For example,utilizing testing system controller logic 214 may include instructions regarding aspects of the video segments such as number of frames, speed, image normalization, or other preprocessing. Alternatively, in some embodiments of system 200, the image data captured by imaging system 235 is processed to generate one or more video segments by a testing sample(s) processing component 270, described below. The image data generated by imaging system 235, which may include one or more video segments or aspects thereof, such as frames or images, may be stored as tested device data 208 in storage 205, where it may be accessed by other components of system 200.

[0081] Lighting system 245 is operated to illuminate a device undergoing inspection. Embodiments of lighting system 145 comprise one or more light sources. In some embodiments, lighting system 145 includes light sources that can be configured to create a lighting condition optimal for defect detection. For instance, various properties of lighting are controlled to create specific lighting scenarios to facilitate revealing subtle defects that might not be visible under standard lighting conditions. In some embodiments, the lighting conditions are controlled during device inspection in order to approximate the lighting conditions under which defect signatures, used for facilitating defect detection, are generated or under which training data, for a defect detection model, is generated. This similarity in conditions ensures that the testing samples produced arc highly comparable to the training data, thereby enhancing the accuracy and reliability of the defect detection process. As depicted in example system 200, lighting system 245 is controlled by lighting system controller 240, for instance, to control various properties or parameters of the lighting system 245, such as the number of active light sources, angle(s) of incidence, color or temperature, spectrum, band, strobe or strobe rate, and lumen level. One embodiment of lighting system 245 includes a tunable light source and can include one or more light emitting diode (LED)-based lighting elements that are tunable to specific parameters, such as those described above.

[0082] Device manipulator 255 is operated to control a motion and / or positioning of a device undergoing inspection. For example, device manipulator 255 comprises one or more mechanisms, such as a robotic arm, conveyor belt, roller, vibrating plane, rotating disc, or moving fixture that is capable of producing controlled motions of a device, such as rotational, linear, tilt, or planar motions, or combinations thereof. Some embodiments of device manipulator 255 operate in coordination with imaging system 235 and lighting system 245, under the control of device manipulator controller 250, to provide synchronized or coordinated movement of the tested device during image capture. Such synchronization can facilitategenerating video segments that capture the device from particular angles or conditions, which in some instances can be caused to change during generation of a video segment, thereby facilitating the detection of defects that may only be apparent during specific movements or from certain perspectives.

[0083] Continuing with FIG. 2, defect detection logic generator 260 is generally responsible for generating the defect detection logic, as described previously. In various embodiments, defect detection logic comprises computer instructions for determining a defect or likelihood thereof, and may include one or more machine learning models or algorithms for the defect determination. For example and as described herein, some instances of defect detection logic comprise computer instructions for determining a similarity between a testing sample feature and a defect signature, which can be used to determine a likelihood of a defect in the tested device. In some instances, defect detection logic includes defect signatures and / or uses a machine learning model trained using labeled training data that includes defect signatures, as described herein. Accordingly, in various embodiments, defect detection logic generator 260 is responsible for generating defect signatures, as described previously, and / or training the one or more defect detection models.

[0084] In some embodiments, defect detection logic generator 260 generates (or facilitates the generation of) video data, to be used as labeled training data for the modcl(s). This process includes capturing image data of both defective and non-defective devices of a particular type under controlled conditions. The controlled conditions are designed to mimic the testing conditions to be applied in a testing environment for detecting specific defects of a device undergoing testing. In particular, this similarity in conditions ensures that the training data more closely resembles the actual testing scenarios, thereby enhancing the accuracy of defect detection.

[0085] Thus in some embodiments, based on the controlled conditions under which the labeled video data (used for training the models or used for defect signatures) is generated, defect detection logic generator 260 also determines computer instructions for controlling testing conditions of a testing environment. These instructions can include controlling aspects of lighting conditions, device motion, image-capture configurations, or other relevant parameters of conditions in which training video data was generated for use by defect detection models or for defect signatures.

[0086] In some implementations, defect detection logic generator 260 is responsible for determining the optimal controlled conditions for generating labeled video data for use intraining a model or as a defect detection signature. For example, based on particular defect types being represented in the labeled video data, certain lighting, motions, or image capturing properties are configured to maximize a visual indication of a defect across one or more frames of the video data. In one example, the angle of incidence of lighting and the motion of the device are controlled to highlight or emphasize a discoloration on the device, indicating a likely defect.

[0087] In some instances, the controlled conditions for generating labeled video data are predetermined based on the device type and / or defect type, or they may be updatable as new information is learned about optimal conditions for capturing particular defect types. Accordingly, defect detection logic generator 260 determines (or otherwise provides) computer instructions for controlling testing conditions in the testing environment, to ensure that testing conditions in the testing environment closely match those used for generating defect signatures and / or model training data.

[0088] In regards to training a defect detection model, some embodiments of defect detection logic generator 260 employ supervised learning using labeled video data representing a particular device type with and without specific defect types. In particular, labels indicate whether each video data sample contains the particular defect type or not. The specific training algorithm utilized by defect detection logic generator 260 depends on the particular model being used. It is contemplated that any suitable model training approach can be utilized, as would be known by a person skilled in the art of training models using labeled video data. For example, in embodiments described herein using a Siamese Neural Network architecture, the models may be trained using either a contrastive loss function or triplet loss. In the contrastive loss approach, pairs of samples are compared with the goal of minimizing the distance between samples of the same class (defective or non-defective) and maximizing the distance between samples of different classes. A triplet loss approach fortraining the model involves training on triplets of samples: an anchor, a positive example (same class as the anchor), and a negative example (different class from the anchor). For instance, where the anchor is a video sample representing a particular defect, the positive example is another video sample representing the same particular defect, and the negative example is a different video sample representing the same device, but without the defect. The model learns to push the anchor closer to the positive example and farther from the negative example in the feature space.

[0089] In some embodiments, the defect detection logic generated by defect detection logic generator 260 includes instructions for identifying regions of interest (ROIs) on a testeddevice and detecting data features associated with these ROIs that may indicate possible defects. These instructions may be predetermined based on the device type and / or defect type, or they may be dynamically updated or modified, as described herein, based on learnings from previously detected defects. For instance, if a particular defect is detected in a specific region of a device, the defect detection logic may be programmatically updated or modified to be more sensitive to that region in future inspections of similar devices.

[0090] As shown in the example system 200, the defect detection logic that is determined by defect detection logic generator 260 may be stored as defect detection logic 218 in a device profile 210, which may be associated with a particular device type. For implementations where defect detection logic generator 260 produces computer instructions for controlling testing conditions, these instructions may be stored as "testing system controller(s) logic 214" within device profile 210. This organization provides that relevant information for defect detection, including the logic itself and the instructions for creating appropriate testing conditions, is available for each specific device type.

[0091] Testing sample(s) processing 270 is generally responsible for processing a testing sample to determine one or more testing sample features, for use by defect detector 280 to determine likelihood of a defect in the tested device. As described herein, a testing sample feature may comprise a data feature, such as a feature vector, determined from the testing sample. Alternatively or in addition, a testing sample feature includes at least a portion of the video segment of a testing sample.

[0092] Testing sample(s) processing 270 receives image data for a tested device undergoing defect inspection, which may be received from imaging system 235. The imaging data may comprise a video segment representing the tested device, or aspects thereof, such as frames or images. In some embodiments, information is also received regarding the particular device type represented in the image data. In some embodiments, information about the device is received from device data 212 in a device profile 210 corresponding to the type of device that is undergoing inspection. In some embodiments where a testing sample feature includes at least a portion of the video segment, testing sample(s) processing 270 prepares the video segment from the received imaging data, which may comprise preprocessing the received image data, combining image data from multiple cameras, or pruning imaging data to exclude irrelevant portions of video. In some implementations, testing sample(s) processing 270 processes the imaging data in stages. For example, it may first perform a preliminary analysisto identify regions of interest in the device, and then target those regions for further processing to extract (or otherwise determine) a testing sample feature.

[0093] As shown in example system 200, testing sample(s) processing 270 comprises subcomponents including preprocessing 272 and feature detection / extraction 274. Preprocessing 272 is generally responsible for preparing the imaging data of the tested device for one of the feature detection and extraction functions employed by feature detection / extraction 274. Some embodiments of preprocessing 272 include, for example and without limitation, normalizing the image data, performing image processing on the image data, such as modifying contrast, color, sharpness, or similar image properties, and / or pruning video by removing frames that do not show relevant aspects of the tested device. Some embodiments of preprocessing 272 combine image data from multiple sources; for instance video data from multiple sources can be combined into a single video segment by concatenation. Similarly, some embodiments of preprocessing 272 process image data by dividing video frames into multiple segments or regions based on boundaries of objects, such as specific aspects of the tested device to facilitate detection of specific types of defects. In some instances, the image data is preprocessed by normalizing spatial data and transforming temporal data, such as data from frames, into a suitable sequence format.

[0094] Feature detection / extraction 274 is generally responsible for determining the testing sample features using the image data, which is preprocessed in some instances. As described herein, a testing sample feature can represent an aspect of a tested device across multiple frames of a video segment, and may comprise a spatio-temporal data feature. Some embodiments of feature detection / extraction 274 utilize feature detection / extraction logic 216 to determine a testing sample feature. Feature detection / extraction logic 216 comprises computer instructions and may further include rules, conditions, associations, models, or other criteria for programmatically detecting or extracting features from the testing sample image data of a tested device. Feature detection / extraction logic 216 may take different forms depending on the particular testing sample feature to be determined and / or the imaging data.

[0095] In some embodiments, feature detection / extraction logic 216 includes logic to identify a spatial data feature corresponding to a region of interest (ROI) on an image of a tested device, such as a first frame of a video segment. The logic detects the data feature in the ROI across at least one additional frame. In some implementations, an ROI is detected in the frames using pixel segmentation or aspects thereof, such as edge detection or thresholding. In some embodiments of feature detection / extraction logic 216, an optical flow technique isemployed to compute a motion of a feature or an object between consecutive frames, capturing temporal information about movement. Thus, feature detection / extraction 274 can utilize these motion vectors as additional feature data alongside spatial feature data extracted from individual frames, thus providing a spatio-temporal feature as a testing sample feature. In some instances, the object that is tracked between frames is the tested device or an aspect of the tested device, such as an image artifact detected from a particular frame of the video segment.

[0096] In some embodiments, feature detection / extraction logic 216 includes one or more object tracking models to determine a testing sample feature from a video segment representing a tested device. In some instances, the object tracking model is utilized by determining an image artifact or a region of interest regarding an aspect of the tested device that is depicted in a first frame of the video segment, and tracking the artifact or region of interest across multiple frames. Various object tracking models may be utilized, such as ByteTrack, Simple Online and Realtime Tracking (SORT), or Deep Simple Online and Realtime Tracking (DeepSORT).

[0097] In some embodiments, feature detection / extraction logic 216 includes a 3D CNN, RNN, and / or a video transformer for determining a testing sample feature from the video segment. For implementations employing an RNN, the RNN may comprise a Convolutional Long Short-Term Memory (ConvLSTM) or an Inflated 3D ConvNct (I3D). These models arc utilized to process sequential data and can extract spatio-temporal features from video segments. For example, ConvLSTM combines convolutional operations with LSTM units to capture both spatial and temporal dependencies in the data. The I3D model inflates 2D CNN architectures into 3D, allowing it to learn spatio-temporal features directly from video data.

[0098] In some embodiments, feature detection / extraction logic 216 comprises a model that is trained to determine spatio-temporal features associated with a particular type of defect. Moreover, in some embodiments a model from defect detection logic 218 is utilized. (That is, feature detection / extraction logic 216 includes instructions to use a model from defect detection logic 218 or to train a model as described in connection with defect detection logic 218.) Thus, the model is trained to detect features relevant to defect detections, such as described in connection with defect detection logic generator 260. For example, the model is trained under a supervised learning approach using labeled video data representing a device having the particular type of defect.

[0099] These example models and techniques of feature detection / extraction logic 216 are non-limiting. It is contemplated that any number of suitable models or techniques areemployed by feature detection / extraction 274 to determine the testing sample feature, as described herein. The testing sample features determined by testing sample(s) processing 270 may be stored as tested device data 208 in storage 205, where it may be accessed by defect detector 280.

[0100] Defect detector 280 is generally responsible for determining a defect in a device. In particular, embodiments of defect detector 280 determine a defect, which can include determining a likelihood of the defect or predicting a likelihood of a future failure of the device that is undergoing inspection. The defect (or likelihood thereof) may be determined based on the testing sample features determined by testing sample(s) processing 270.

[0101] In particular, some embodiments of defect detector 280 receive input data from testing sample(s) processing 270, which may include testing sample features or processed video segments representing the device under inspection. The testing sample features then may be processed to identify potential defects or anomalies. In some implementations, defect detector 280 determines a defect using defect detection logic 218 with one or more testing sample features or an aspect of the testing sample features. Some embodiments of defect detection logic 218 comprise defect detection logic 118, described in connection with Fig. 1. Additional details of defect detection logic 218 are described in connection with defect detection logic 118 of FIG. 1 and defect detection logic generator 260 of FIG. 2.

[0102] Further, as described in connection with defect detection logic 118 of FIG. 1, some embodiments of defect detection logic 218 include one or more machine learning models, such as CNNs, RNNs, video transformers, or other suitable architectures for processing spatiotemporal data. Accordingly, these models are trained to recognize patterns or features associated with various types of defects. Some embodiments of defect detection logic 218 include computer instructions for performing a comparison between a testing data feature and a defect signature, and for determining a measure of similarity such as a distance measurement or other statistical measure, which may be used to represent a likelihood of a defect. Additional details and embodiments of defect detection logic 218 are described in connection with defect detection logic 118 of FIG. 1 and defect detection logic generator 260 of FIG. 2.

[0103] In some embodiments, defect detector 280 provides an indication of whether or not the tested device has a defect and may further provide a likelihood of the defect. Defect detector 280 may also provide an indication of the defect type that is detected. The output of defect detector 280 may be provided to operator / clinician UI 220, where it may be used to generate a notification. For instance, the notification indicates the likelihood of a defect for theparticular tested device, and may further indicate information about the particular defect. Additional details regarding embodiments of this notification are described in connection with operator / clinician UI 220 and FIG. 1.

[0104] Example system 200 of FIG. 2 also includes storage 205. Storage 205 generally stores information including data, computer instructions (for example, software program instructions, routines, or services), logic, profiles, and / or models used in embodiments described herein. In an embodiment, storage 205 comprises a data store (or computer data memory). Further, although depicted as a single data store component, storage 205 can be embodied as one or more data stores or in the cloud. In one embodiment, storage 205 is embodied as memory 712 of computing device 700 of FIG. 7.

[0105] As shown in example system 200 of FIG. 2, storage 205 includes tested device data 208 and one or more device profiles 210. Tested device data 208 is described previously in connection with testing sample(s) processing 270. Device profile 210 is one example of a device profile, which as described herein, corresponds to a particular device or device type undergoing defect inspection. In particular, a device profile comprises a collection of data and instructions that facilitate the testing process and defect detection for a specific device type.

[0106] The example device profile 210 depicted in system 200 includes device data 212, which includes information about the particular device or device type corresponding to the device profile. For example, device data 212 may include manufacturing information about the device, such as a serial number or identification number, an indication of a manufacturing location, a lot number, or a run number. Device data 212 may be used to generate notifications, such as a report indicating a detected defect or no defect detected for a particular device. For instance, some embodiments of operator / clinician UI 220 generate a notification comprising a report that includes information about the device type, which is received from device data 212. Example device profile 210 also includes testing system controller logic 214, feature detection / extraction logic 216, and defect detection logic 218, each of which has been described previously.

[0107] Turning now to FIGS. 3 A and 3B, aspects are illustratively shown of an example of a device type for which embodiments of this disclosure may be used for defect detection. Here, the example comprises a stent, which is a medical device. Stents are one example of devices that are susceptible to various types of defects. Commonly used to support weakened or narrowed blood vessels, stents are typically made from biocompatible materials such asmetals or polymers. However, during the manufacturing process or through handling, defects can occur that impact the performance and safety of the device.

[0108] Referring to FIG. 3A, an example stent 301 is depicted. Due to the small size and intricate structure of stent 301 , human inspection of defects can be particularly challenging. Moreover, the handling required for inspection may itself lead to further damage, underscoring the need for advanced inspection technologies described herein to ensure the integrity of stents and other similarly delicate medical devices.

[0109] Referring now to FIG. 3B, aspects of example defects 310 that can occur in stents are depicted. These include a break defect 312, which may occur due to the brittleness of certain stent materials, particularly after processes like heal treatment or surface finishing. A discoloration defect 314 is also shown, which can indicate issues with material composition or processing. Additionally, a foreign material (FM) defect 316 is illustrated, representing the presence of foreign material or contaminants that can interfere with the stent's function. Other common defects not shown in FIG. 3B but that can occur in stents include burrs or spurs on the surface, weak areas that reduce structural integrity, porosity, delamination, blistering, corrosion, misalignment, pitting, dents, scratches, and warpage.

[0110] It's important to note that while FIGS. 3 A and 3B depict a stent, embodiments of this disclosure arc applicable to many other types of devices, as described previously, including non-medical devices.

[0111] Referring now to FIG. 3C, with continuing reference to FIGS. 3A and 3B, an example video segment 350 is depicted. Video segment 350 represents an example of a testing sample, as described herein and in connection with FIGS. 1 and 2. In one embodiment, video segment 350 may be generated by imaging system 235 or testing sample(s) processing 270 of FIG. 2. The testing sample depicted in video segment 350 comprises image data of a stent 360, which serves as the tested device in this example.

[0112] Video segment 350 comprises multiple frames, with three example frames shown enlarged: frame 351, frame 352, and frame 353. These frames are captured at different times within video segment 350, specifically at times Tl, T2, and T3, respectively. Each of frames 351, 352, and 353 presents a different aspect of stent 360, as the video segment 350 is generated while stent 360 is moved relative to the camera (such as imaging system 235 in FIG.2). Consequently, each frame offers a slightly different perspective of stent 360.

[0113] Notably, each of frames 351, 352, and 353 exhibits at least one example of a defect, with these defects appearing at different times throughout video segment 350. Frame351 displays three example defects: break 362a, discoloration 364a, and FM 366a. Frame 352 shows the same three defects from a different perspective due to the slight movement of stent 360: break 362b, discoloration 364b, and FM 366b. Frame 353 reveals only two of the example defects, again from a different perspective as stent 360 has moved further: break 362c, which is barely noticeable, and discoloration 364c, which is also much less apparent than its appearance in frame 351 as discoloration 364a. The FM defect (identified as FM 366a in frame 351 and FM 366b in frame 352) is not visible in frame 353.

[0114] The example video segment 350 provided in FIG. 3C, particularly in illustrating the three example frames 351, 352, and 353, demonstrates that certain defects may be more accurately detected when multiple frames are utilized. This aligns with certain embodiments described in connection with FIGS. 1 and 2, where testing sample features are determined using multiple frames, referred to as spatio-temporal data features. In particular, FIG. 3C illustrates that in frame 353, only two defects (break 362c and discoloration 364c) are apparent and barely detectable, while the FM defect is not visible. However, as stent 360 undergoes motion and / or lighting conditions are altered, the defects may become more visible in other frames. For example, in frame 351, all three defects (break 362a, discoloration 364a, and FM 366a) are apparent. This variation in defect visibility across frames underscores the potential advantages of utilizing multiple frames for defect detection in some implementations.

[0115] Moreover, certain embodiments of the detection technology described herein are superior to conventional defect detection systems because not only are they capable of detecting a defect in a single frame but these technologies can also leverage spatio-temporal data features indicating defects. For example, consider how the discoloration defect (364a, 364b, and 364c) appears as stent 360 is moved (e.g., rotated). The spatio-temporal data feature can capture how the appearance of this example discoloration defect changes as it moves. This spatio-temporal data feature can be processed using defect detection logic (e.g., defect detection logic 118 or defect detection logic 218, as described in connection with FIG. 1 and FIG. 2, respectively), such as by matching it against a defect signature or applying it to a model trained on video segments of defective and non-defective versions of the device. In this way, the discoloration defect can be more accurately detected and classified than conventional detection technologies or human inspection.

[0116] Turning now to FIGS. 4, 5, and 6, aspects of example process flows 400, 500, and 600 are illustratively depicted for some embodiments of the disclosure. Embodiments of process flows 400, 500, and 600 each comprise a method (sometimes referred to herein asmethods 400, 500 and 600) carried out to implement various example embodiments described herein. Each block or step of methods 400, 500, and 600, as well as other methods described herein, comprises a computing process performed using any combination of hardware, firmware, and / or software. For instance, various functions are carried out by a processor executing instructions stored in memory, such as memory 712 as described in FIG. 7 and / or as storage 205 as described in FIG. 2. Embodiments of these methods can also be implemented using computer-usable instructions stored on computer storage media. Embodiments of the methods are provided by a stand-alone application, a service or hosted service (stand-alone or in combination with another hosted service), or a plug-in to another product, to name a few. For example, the blocks of methods 400, 500, and 600 correspond to operations (or steps) to be performed (as opposed to information to be processed or acted on). In some embodiments, these operations are carried out by one or more computer applications or services, which operate on one or more: computing devices (such as computing system 111 of FIG. 1 or computing device 700 of FIG. 7), or distributed or cloud computing systems (such as described in connection with FIG. 8). Further and in some embodiments, the functions performed by the blocks of methods 400, 500, and 600 are carried out by components of system 200, as described in FIG. 2.

[0117] With reference to FIG. 4, aspects of example method 400 are illustratively provided for detecting or predicting defects in a device, which in some instances comprises a medical device, such as stent 360 described in connection with FIG. 3C.

[0118] At a block 410, method 400 includes receiving a device profile corresponding to a device, the device profile including instructions for controlling testing conditions for generating a testing sample. The device corresponding to the device profile in block 410 is undergoing inspection. In some embodiments, the instructions of the device profile for controlling testing conditions include instructions for controlling the imaging system to generate the testing sample. In some aspects, the imaging system includes a camera, and the instructions for controlling the imaging system comprise instructions specifying at least one of a frame rate for the camera, a shutter speed, a minimum or a maximum number of frames to capture for the video segment, and a duration of video to capture for the video segment. Some implementations further comprise a lighting system, and the instructions of the device profile for controlling testing conditions include instructions for controlling the lighting system. In certain implementations, the instructions for controlling the lighting system comprise instructions specifying at least one of a lighting color temperature, lighting strobe frequency,lumen level, and angle of incidence with regard to the device. In some embodiments, the device profile includes a defect detection logic for use in block 450 for detecting or predicting a defect in the particular device corresponding to the device profile.

[0119] Embodiments of block 410 may earned out by the controller components 230, 240, or 250 of FIG. 2. Where the device profile includes detection logic, generating the detection logic may be performed by defect detection logic generator 260 of FIG. 2. Additional details regarding embodiments of block 410 are described in connection with FIG. 2.

[0120] At a block 420, method 400 includes programmatically controlling a device manipulator to cause a first motion of the device. Embodiments of block 420 programmatically control a device manipulator based on the instructions of the device profile received at block 410. The first motion may comprise a rotational motion, a linear motion, a tilt motion, or a planar motion. In certain implementations, block 420 further comprises programmatically controlling the device manipulator to cause a second motion of the device that is different than the first motion. Embodiments of block 420 may be performed by a device manipulator controller 250 of FIG. 2. Additional details regarding embodiments of block 420 are described in connection with FIG. 2 and device manipulator 155 of FIG. 1.

[0121] At a block 430, method 400 includes generating a testing sample that comprises a video segment representing at least a portion of the device. Embodiments of block 430 generate the testing sample by capturing image data of the device during the first motion and / or any additional motions. Embodiments of block 430 may be carried out by imaging system 235 or testing sample(s) processing 270 of FIG. 2. Additional details regarding embodiments of block 430 are described in connection with FIG. 2 and imaging system 135 of FIG. 1.

[0122] At a block 440, method 400 includes processing the video segment to determine a testing sample feature representing an aspect of the device. Embodiments of block 440 process a video segment of the testing sample determined at block 430 to determine one or more testing features. In some embodiments, including the embodiment shown in method 400 of FIG. 4, the testing feature determined at block 440 represents the device across a plurality of frames. In particular, in some embodiments the testing feature is a spatio-temporal data feature.

[0123] In various implementations, processing the video segment at block 440 to determine the at least one testing sample feature comprises using at least one of: optical flow, an object tracking model, a three-dimensional Convolutional Neural Network (3D CNN), a recurrent neural network (RNN), and a video transformer. In certain implementations, opticalflow is utilized by preprocessing at least a portion of the plurality of frames and computing a motion vector between at least two frames of the plurality of frames. The preprocessing may be carried out by preprocessing 272 subcomponent of testing sample(s) processing 270 in FIG.2. In certain implementations, an object tracking model is utilized by determining a region of interest in a first frame of the plurality of frames, and tracking the region of interest across multiple frames of the plurality of frames.

[0124] In certain implementations, the 3D CNN, RNN, or video transformer comprises a model that is trained to determine at least one spatio-temporal feature associated with a particular defect. As described herein, the training may be performed using labeled video data representing a defective device that is generated by capturing image data of the defective device under a controlled condition that is similar to the testing conditions specified in the device profile received at block 410. For example, the conditions may include a controlled motion similar to the first motion of the testing condition at block 420. In some aspects, processing the video segment to determine the at least one testing sample feature uses an RNN that is a Convolutional Long Short-Term Memory, or uses 3D CNN that is an Inflated 3D ConvNet.

[0125] Embodiments of block 440 may be carried out by testing sample(s) processing 270 of FIG. 2. Additional details regarding embodiments of block 440 are described in connection with FIG. 2, and in particular testing samplc(s) processing 270 and its subcomponents.

[0126] At a block 450, method 400 determines a defect (or likelihood of a defect) associated with the device. Embodiments of block 450 determine a defect (or a likelihood of a defect) associated with the device based on the testing sample feature determined at block 440. In some embodiments, the likelihood of a defect associated with the device is determined using defect detection logic. For example, some embodiments apply the defect detection logic to one or more testing sample features determined at block 440.

[0127] In certain implementations, the defect detection logic comprises a machine learning model trained using a defect signature corresponding to the device, and the likelihood of a defect associated with the device represents a similarity between a testing sample feature and the defect signature. In some aspects, applying the defect detection logic to the testing sample feature set comprises determining a similarity of the testing sample feature set to a defect signature corresponding to the device. The similarity may be determined based on a binary-match decision or a feature similarity distance of the testing sample feature and the defect signature. For instance if the distance is within (or satisfies) a similarity threshold, thensimilarity is determined, and thus the tested device likely has the defect associated with the defect signature.

[0128] In certain embodiments, the defect signature comprises one or more spatiotemporal features associated with one or more defects. The defect signature may be determined from a model trained using video data representing a defective device and a non-defective device. The video data may be generated by capturing image data of the defective device and the non-defective device under a controlled condition that is similar to the testing condition specified in the device profile received at block 410.

[0129] Some implementations of block 450 further comprise determining a type of defect that characterizes the determined defect (or likely defect). Based on the determined type of defect, the detection logic can be updated to have increased sensitivity for detecting that type of defect. For example, the device detection logic can be updated by designating or adding a region of interest (ROI) for determining the testing sample feature. The ROI includes a location on the device corresponding to the determined defect. The device detection logic can also be updated by updating an existing ROI for determining the testing sample feature, the existing ROI including the location on the device corresponding to the determined defect. In some implementations, the existing ROI is updated by adding or adjusting a weighting associated with the testing sample feature determined from the ROI; updating a coefficient of a defect detection model that is included in the defect detection logic, the coefficient corresponding to a data feature associated with the determined defect; updating a defect detection model that is included in the defect detection logic by fine-tuning the model based on the determined defect; or adjusting a similarity threshold of the defect detection logic.

[0130] Embodiments of block 450 may be carried out by defect detector 280 of FIG. 2. Additional details regarding embodiments of block 450 are described in connection with FIG.2, and in particular with defect detector 280.

[0131] At a block 460, method 400 includes providing a notification regarding the defect associated with the device. Embodiments of block 460 provide a notification based on the likelihood of a defect determined at block 450. For example, the notification may indicate a likely defect is detected or no defect is detected, such as described in connection with items 170 and 180 of FIG. 1. In some embodiments, the notification regarding the defect includes an indication of the determined likelihood of the defect, an indication of a defect type, and / or manufacturing information regarding the device. Embodiments of block 460 may be carriedout by operator / clinician UI 220 of FIG. 2. Additional details regarding embodiments of block 460 are described in connection with FIG. 2 and block 185 of FIG. 1.

[0132] With reference to FIG. 5, aspects of example method 500 are illustratively provided for detecting or predicting defects in a device, which in some instances comprises a medical device, such as stent 360 described in connection with FIG. 3C. In particular, method 500 includes a computer-implemented method for processing particular image data, such as video data, of the device to determine a potential defect by employing particular image processing or machine learning. Some embodiments of method 500 include determining or using spatio-temporal features extracted from video data representing the device to enhance defect detection functionality.

[0133] Accordingly, at a block 510, method 500 includes accessing or receiving a video segment representing at least a portion of a device. For example, the video segment may comprise a plurality of frames representing a testing sample corresponding to the device, as described herein. The video segment may be accessed by a computing system or computer processor that is performing the operations of method 500, such as computing device 700 of FIG. 7, or more specifically by processor(s) 714 of computing device 700. Further, the video segment may be accessed at block 510 from an imaging system, such as imaging system 235 described in FIG. 2, or from data storage, such as tested device data 208 in storage 205, also described in FIG.2.

[0134] In some embodiments, the video segment accessed at block 510 comprises a plurality of video frames. In some instances, the video segment is generated by capturing, using a camera, image data of the device while controlling a first motion comprising a motion of the device with respect to the camera or controlling a motion of the camera with respect to the device. The first motion may comprise a rotational motion, linear motion, tilt motion, or planar motion. In some cases, the first motion is determined based on the particular type of the device being inspected.

[0135] Embodiments of block 510 may be carried out by testing sample(s) processing 270 of FIG. 2. Additional details regarding embodiments of block 510 are described in connection with FIG. 2, and in particular with testing sample(s) processing 270 and its subcomponents.

[0136] At a block 520, method 500 includes processing the video segment accessed at block 510 to determine one or more testing sample features. The testing sample feature(s) represent an aspect of the device, and may be referred to herein as a testing sample feature set.In some embodiments, the video segment comprises a plurality of frames that are processed to determine a particular sample feature, which represents, across the plurality of frames, the aspect of the device being inspected. For instance, the particular sample feature may comprise a spatio-temporal feature.

[0137] Some embodiments of block 520 include processing the video segment to determine the at least one testing sample feature by using an optical flow algorithm, an object tracking model, a three-dimensional Convolutional Neural Network (3D CNN), a recurrent neural network (RNN), and / or a video transformer. In some instances, processing the video segment uses an RNN that is a Convolutional Long Short-Term Memory, or processing the video segment uses 3D CNN that is an Inflated 3D ConvNet, as described herein in connection with FIGS. 1 and 2.

[0138] Some embodiments of block 520 that use optical flow include preprocessing at least a portion of the plurality of frames and computing a motion vector between at least two frames of the plurality of frames. Alternatively, an object tracking model is utilized at block 520 by determining a region of interest in a first frame of the plurality of frames of the video segment, and tracking the region of interest across multiple frames of the plurality of frames. Alternatively, still, some embodiments of block 520 that utilize a 3D CNN, RNN, or video transformer include using model that is trained to determine at least one spatio-temporal feature associated with a defect. For example, as described herein, training may be performed using labeled video data representing a defective device and representing a non-defective device. In some instances, at least one testing sample feature comprises a first spatio-temporal feature associated with the particular defect or type of defect that is detected or predicted in method 500.

[0139] Embodiments of block 520 may be carried out by testing sample(s) processing 270 of FIG. 2. Additional details regarding embodiments of block 520 are described in connection with FIG. 2, and in particular with testing sample(s) processing 270 and its subcomponents.

[0140] At a block 530, method 500 includes determining a defect associated with the device by applying a defect detection logic to the testing sample feature(s) determined at block 520. As further described herein, defect detection logic comprises computer instructions for determining a defect or likelihood thereof, and may include one or more machine learning models or algorithms for the defect determination. Accordingly, block 530 may include determining a likelihood that the device has an existing defect or predicting a likelihood thatthe device will fail during its expected operational lifetime. Some embodiments of block 530 further include determining a type of the defect that characterizes the particular determined defect. For example, various types of defects of a stent device are described in connection with FIG. 3B. In some instances, the defect detection logic applied at block 530 corresponds to the particular type of device being inspected for defects in method 500, such as described in connection with defect detection logic generator 260 of FIG. 2.

[0141] In some embodiments of block 530, the defect detection logic comprises a machine learning model trained using a defect signature corresponding to the device type. In some instances, this defect signature comprises at least one spatio-temporal feature associated with the particular defect that is determined at block 530. Accordingly, in some implementations of block 530, the defect is determined based on a similarity of the testing sample feature(s) and the defect signature. The similarity may be determined, for instance, based on a binary-match decision using a model of the detection logic or based on a feature similarity distance, between the testing sample feature(s) and the defect signature, satisfying a similarity threshold.

[0142] Embodiments of block 530 may be carried out by defect detector 280 of FIG. 2. Additional details regarding embodiments of defect detection logic are described in connection with defect detection logic generator 260 and defect detection logic 218 of FIG. 2. Additional details regarding embodiments of the defect detection operation of block 530 are described in connection with FIG. 2, and in particular defect detector 280 and defect detection logic 218.

[0143] At a block 540, method 500 includes providing an indication of the defect associated with the device. In some embodiments of block 540, the indication regarding the defect comprises at least one of a likelihood of the defect, a type of defect, and manufacturing information regarding the device. The operations of block 540 may be carried out by operator / clinician user interface, such as operator / clinician UI 220 of FIG. 2. Additional details regarding embodiments of block 540 are described in connection with FIG. 2, and in particular operator / clinician UI 220.

[0144] Some embodiments of method 500 further include causing the device determined likely to have a defect to be segregated for: disposal, further testing, or repair. For instance, the device may be marked for or placed in a designated storage location for disposal, further testing, or repair.

[0145] In some embodiments of method 500, where a first tested device is determined at block 530 as likely having a particular defect, method 500 includes programmaticallyupdating the defect detection logic to be more sensitive towards detecting the particular defect when testing other devices of the same or similar device type as the first tested device. For example, the defect detection logic is updated based on learnings from a detected defect. In particular, in some instances where a particular device is determined to have a particular type of defect, other devices of the same type may have a greater likelihood of having that same (or similar) type of defect. Accordingly, these embodiments of the defect detection system are programmatically adaptive and enabled to learn to better detect particular defects based on prior detections.

[0146] In particular, in some of the implementations where block 530 includes determining a type of the defect that characterizes the device defect, method 500 further includes updating the device detection logic. For example and as described herein, the device logic can be updated to have increased sensitivity for detecting the particular type of defect when testing similar devices. In some instances, the device detection logic is updated by performing at least one of: designating or adding a region of interest (ROI) for determining the testing sample feature(s) at block 520, the ROI including a location on the device corresponding to the determined defect; updating an existing ROI for determining the testing sample feature(s), the existing ROI including the location on the device corresponding to the determined defect, the existing ROI updated by adding or adjusting a weighting associated with the testing sample feature determined from the ROI; updating a coefficient of a defect detection model that is included in the defect detection logic, the coefficient corresponding to a data feature associated with the determined defect; updating a defect detection model that is included in the defect detection logic by fine-tuning the model based on the determined defect; and / or adjusting a similarity threshold of the defect detection logic. Additional details regarding embodiments of updating device detection logic are described in connection with block 450 of method 400 and defect detector 280 of FIG. 2.

[0147] With reference to FIG. 6, aspects of example method 600 are illustratively provided for detecting or predicting defects in a device. In particular, method 600 includes computer-performed operations for generating a device profile for a particular device or type of device to be inspected. The example device profile generated according to method 600 includes defect detection logic and computer- or machine-readable instructions for reproducing in a testing environment the controlled conditions present when generating a defect signature and / or training data for a defect detection model, either of which is a part of the device detection logic. In some embodiments, at least a portion of the operations of method 600 are carried outby defect detection logic generator 260 of FIG. 2. Additional details regarding embodiments of each step of method 600 are described in connection with FIG. 2, and in particular defect detection logic generator 260.

[0148] Accordingly, at a block 610, method 600 includes generating video data comprising a plurality of video segments captured under controlled conditions. The plurality of video segments includes at least a first video segment representing a defective instance of a device and a corresponding label indicating the particular defect, and a second video segment representing a non-defective instance of the same type of device with a corresponding label indicating no defect. Additionally, each video segment of the plurality comprises multiple video frames. Further, the video of each segment is captured under a controlled condition, such as a first controlled motion of the defective and non-defective instances of the particular device with respect to the camera (or of the camera with respect to the defective and non-defective instances of the device). In some implementations, the controlled conditions include conditions specifying generating the video data according to a configuration of an imaging system that includes at least one camera. Similarly, in some implementations, the controlled conditions include a first lighting condition present when capturing the video segments representing the defective and non-defective instances of the device.

[0149] In some embodiments of block 610, the plurality of video segments further includes a first plurality of video segments including the first segment, and referred to herein as "defective device training data," and a second plurality of video segments including the second segment, and referred to herein as "non-defective device training data.” In some instances, each video segment of the first plurality of video segments is generated by capturing video of a different defective instance of the device having the particular defect and having associated therewith a corresponding label indicating the particular defect. Similarly, each video segment of the second plurality of video segments may be generated by capturing video of a different, non-defective instance of the device, and having associated therewith a corresponding label indicating no defect.

[0150] At a block 620, method 600 includes training a defect detection model to detect defect features corresponding to a particular defect. In particular, block 620 includes training the defect detection model using the plurality of video segments and their corresponding labels. Thus the particular defect that the defect detection model is trained to detect is the defect represented in the defective device training data. In some implementations, the defect detectionmodel comprises a three-dimensional Convolutional Neural Network (3D CNN), a Convolutional Long Short-Term Memory (ConvLSTM), or a video transformer.

[0151] In certain implementations, the defect detection model at block 620 comprises a Siamese 3D CNN, and training the defect detection model is performed using positive pairs of video segments and negative pairs of video segments. In these implementations, the positive pairs of video segments include pairs of video segments from the defective device training data (i.e., the first plurality) and pairs of video segments from the non-defective device training data (i.e., the second plurality). The negative pairs of video segments comprise a video segment from the defective device training data (i.e., the first plurality) and a video segment from the non-defective device training data (i.e., the second plurality) of video segments.

[0152] Alternatively, in certain implementations, training the Siamese 3D CNN defect detection model is performed using triplet loss by providing a plurality of three- video-segment inputs to the defect detection model. In particular, two of the three video segments provided as inputs, for training the defect detection model, comprise two instances of the defective device training data or two instances of the non-defective device training data, and the third video segment input is different than the training data of the first two video segment inputs. Thus, for example, the three-video-segment input comprises two segments from the defective device training data and a third segment from the non-defective device training data. Alternatively, the three-video-segment input comprises two segments from the non-defective device training data and a third segment from the defective device training data.

[0153] At a block 630, method 600 includes generating defect detection logic for detecting the particular defect associated with the device. In some implementations, the defect detection logic generated at block 630 comprises a defect signature that includes the defect detection model trained at block 620, and further includes machine-readable instructions for applying, to the trained defect detection model, a testing sample feature corresponding to an instance of the device undergoing inspection. In other implementations, the defect detection logic generated at block 630 comprises a defect signature that includes one or more spatiotemporal features corresponding to the particular defect and determined from the trained defect detection model. The defect detection logic may further include machine-readable instructions for computing a similarity of the defect signature to a testing sample feature corresponding to an instance of the device undergoing inspection.

[0154] At a block 640, method 600 includes generating machine-readable instructions for reproducing the controlled conditions. In particular, embodiments of block 640 generatemachine-readable instructions for reproducing, in a device testing environment, the controlled conditions under which the video segments, representing defective and non-defective device training data, were generated at block 610.

[0155] Accordingly, in various implementations, the machine-readable instructions generated at block 640 can include: instructions for configuring or operating an imaging system of the testing environment, based on the configuration of the imaging system utilized at block 610; instructions for controlling a device manipulator to cause a controlled motion of an instance of the device undergoing inspection, based on a controlled motion applied in certain embodiments at block 610; instructions for controlling a lighting condition of a testing environment, based on the configuration of lighting conditions applied in certain embodiments of 610; instructions for controlling other aspects of the testing environment, such as temperature, humidity, electric or magnetic fields, or other conditions of the testing environment, based on the configurations of these conditions when the plurality of video segments were generated at block 610, or any combination of these.

[0156] At a block 650, method 600 includes generating a device profile corresponding to the device by associating an indication of the device with the defect detection logic and the machine-readable instructions. In particular, embodiments of block 650 generate a device profile that includes at least the defect detection logic generated at block 630 and the machine-readable instructions generated at block 640. In some embodiments, the device profile includes information about the particular device and / or an indication of the particular defect(s) corresponding to the defect detection logic. Additional examples of a device profile are described in connection with various embodiments of device profile 210 of FIG. 2.

[0157] At a block 660, method 600 includes providing the device profile to be accessible for inspecting a candidate instance of the device for the particular defect. In particular, embodiments of block 660 provide the device profile generated at block 650 so that it is accessible when inspecting a candidate instance of the device for a particular defect corresponding to the defect detection logic generated at block 630.

[0158] In some embodiments, the device profile is made available in a database, server, or other data storage, which may be local, remote, or distributed. For instance, the example device profile 210 of FIG. 2 is stored on storage 205 where it is accessible to the other components of system 200, as further described in connection with FIG. 2.

[0159] In some implementations, the device profile is provided to be accessible via image recognition of the device by an imaging system of the testing environment. For example,images taken of the candidate instance of the device may be classified as a particular device type using a vision recognition technique. Once the device is classified as a particular device type, a device profile corresponding to that device type can be accessed from a data store that stores various device profdes, such as storage 205 of FIG. 2.

[0160] In some implementations, the device profile is provided to be accessible via a digital marker on the candidate instance of the device, or on product packaging or labeling associated with the candidate instance of the device. The digital marker corresponds to an address for a particular device profile in a data store. For example, the digital marker may correspond to a uniform resource locator (URL) for the device profile corresponding to the device type. In certain embodiments, the digital marker comprises a quick reference code (QR Code). In some of these embodiments, determining that the candidate instance of the device has the particular defect includes accessing, from a data store, a device profile corresponding to the type of the device, wherein the access is facilitated by scanning (or reading) the QR code using a camera, such as a camera in an imaging system of the testing environment.

[0161] Some embodiments of method 600 further comprise utilizing the device profde generated at block 650 to determine that a candidate instance of the device has the particular defect. For example, this determination may involve accessing a video segment representing at least a portion of the candidate instance of the device that is generated in a testing environment under controlled conditions specified in the device profile. The video segment may be processed to determine one or more testing sample features. The particular defect then can be determined by applying the testing sample feature(s) to the defect detection logic generated at block 630. Based on the determination, a notification can be provided indicating that the candidate instance of the device likely has (or does not have) the particular defect. In some instances, these embodiments are carried out by defect detector 280 of FIG. 2. Additional details and embodiments of using a device profile, such as the device profile generated in method 600, for determining a likely defect are described in connection with FIG. 2.

[0162] Accordingly, various aspects of technology have been described directed to systems and methods for detecting or predicting defects (or a likelihood of a defect) in devices. It is understood that various features, subcombinations, and modifications of the embodiments described herein are of utility and are employed in other embodiments without reference to other features or subcombinations. Moreover, the order and sequences of steps shown in the example methods 400, 500, and 600 are not meant to limit the scope of the present disclosure in any way, and in fact, the steps can occur in a variety of different sequences withinembodiments hereof. Such variations and combinations thereof are also contemplated to be within the scope of embodiments of this disclosure.Other Embodiments

[0163] In some embodiments, a system for detecting or predicting defects in a medical device is provided. The system comprises an imaging system operable to capture image data, a device manipulator operable to control a motion of the medical device, at least one computer processor, and computer memory having computer-readable instructions embodied thereon, that, when executed by the at least one computer processor, perform operations. The operations comprise receiving a device profile corresponding to the medical device, the device profile including instructions for controlling testing conditions for generating a testing sample corresponding to the medical device. The operations further comprise, based on the device profde, programmatically controlling the device manipulator to cause a first motion of the medical device. The operations further comprise causing the imaging system to generate the testing sample comprising a video segment representing at least a portion of the medical device, the video segment generated by capturing image data of the medical device during the first motion, the video segment comprising a plurality of frames. The operations further comprise processing the video segment to determine, using the plurality of frames, at least one testing sample feature representing, across the plurality of frames, an aspect of the medical device, thereby forming a testing sample feature set. The operations further comprise, based on the testing sample feature set, determining a likelihood of a defect associated with the medical device. The operations further comprise based on the determined likelihood, causing a notification, regarding the defect associated with the medical device, to be provided via the system-user interface.

[0164] Advantageously, these and other embodiments, as described herein, improve existing defect detection technologies by providing a more comprehensive and accurate inspection process. The system leverages video-based analysis and machine learning techniques to detect subtle defects that may be missed by conventional static image analysis or human inspection. By utilizing spatio-temporal features extracted from video segments, the system can identify defects that only become apparent under specific conditions or over time. This approach enables testing of every device or a significantly larger sample size, therebyenhancing overall quality control and safety assurance, particularly for critical applications such as medical devices.

[0165] In any combination of the above embodiments, processing the video segment to determine the at least one testing sample feature comprises using at least one of: optical flow, an object tracking model, a three-dimensional Convolutional Neural Network (3D CNN), a recurrent neural network (RNN), and a video transformer.

[0166] In any combination of the above embodiments, optical flow is utilized by preprocessing at least a portion of the plurality of frames and computing a motion vector between at least two frames of the plurality of frames; the object tracking model is utilized by determining a region of interest in a first frame of the plurality of frames, and tracking the region of interest across multiple frames of the plurality of frames; or the 3D CNN, RNN, or video transformer comprises a model that is trained to determine at least one spatio-temporal feature associated with a particular defect, the training performed using labeled video data representing a defective medical device, the labeled video data generated by capturing image data of the defective medical device under a controlled condition that is similar to the testing condition, including at least a controlled motion similar to the first motion.

[0167] In any combination of the above embodiments, processing the video segment to determine the at least one testing sample feature uses an RNN that is a Convolutional Long Short-Term Memory, or uses 3D CNN that is an Inflated 3D ConvNet.

[0168] In any combination of the above embodiments, the device profile includes a defect detection logic for detecting or predicting defects in the medical device.

[0169] In any combination of the above embodiments the likelihood of the defect associated with the medical device is determined by applying the defect detection logic to the testing sample feature set.

[0170] In any combination of the above embodiments, the defect detection logic comprises a machine learning model trained using a defect signature corresponding to the medical device.

[0171] In any combination of the above embodiments, the likelihood of the defect associated with the medical device represents a similarity score corresponding to the similarity of the testing sample feature set and the defect signature.

[0172] In any combination of the above embodiments, the testing sample feature set comprises at least a portion of the video segment and the defect signature comprises video datarepresenting at least one instance of a particular defect occurring in a defective version of the medical device.

[0173] In any combination of the above embodiments the defect detection logic comprises a Siamese Neural Network model configured to determine a similarity between the testing sample feature set and the defect signature.

[0174] In any combination of the above embodiments, applying the defect detection logic to the testing sample feature set comprises determining a similarity of the testing sample feature set to a defect signature corresponding to the medical device, the similarity determined based on a binary-match decision or a feature similarity distance, of the testing sample feature set and the defect signature, satisfying a similarity threshold.

[0175] In any combination of the above embodiments, the defect signature comprises one or more spatio-temporal features associated with defects.

[0176] In any combination of the above embodiments the defect signature is determined from a model trained using video data representing a defective medical device and a non-defective medical device, the video data generated by capturing image data of the defective medical device and the non-defective medical device under a controlled condition that is similar to the testing condition, including at least a controlled motion similar to the first motion.

[0177] In any combination of the above embodiments, the notification regarding the defect includes an indication of the determined likelihood of the defect, an indication of a defect type, and manufacturing information regarding the medical device.

[0178] In any combination of the above embodiments, the instractions of the device profile for controlling testing conditions include instructions for controlling the imaging system to generate the testing sample.

[0179] In any combination of the above embodiments the imaging system is caused to generate the testing sample based on the instractions for controlling the imaging system.

[0180] In any combination of the above embodiments, the imaging system includes a camera.

[0181] In any combination of the above embodiments the instructions for controlling the imaging system comprise instructions specifying at least one of a frame rate for the camera, a shutter speed, a minimum or a maximum number of frames to capture for the video segment, and a duration of video to capture for the video segment.

[0182] In any combination of the above embodiments, the system further comprises a lighting system.

[0183] In any combination of the above embodiments, the instructions of the device profde for controlling testing conditions include instructions for controlling the lighting system.

[0184] In any combination of the above embodiments, the instructions for controlling the lighting system comprise instructions specifying at least one of a lighting color temperature, lighting strobe frequency, lumen level, and angle of incidence with regard to the medical device.

[0185] In any combination of the above embodiments, the first motion comprises a rotational motion, a linear motion, a tilt motion, or a planar motion.

[0186] In any combination of the above embodiments, the operations further comprise programmatically control the device manipulator to cause a second motion of the medical device, the second motion being different than the first motion.

[0187] In some embodiments, a computing system is provided, employing any components of the computerized (or computer, computing, or cloud) system of any of the embodiments described herein. The computing system comprises at least one computer processor, and computer memory having computer-readable instructions embodied thereon, that, when executed by the at least one computer processor, perform operations. The operations comprise accessing a video segment representing at least a portion of the device and comprising a plurality of frames. The operations further comprise processing the video segment to determine, using the plurality of frames, at least one testing sample feature representing, across the plurality of frames, an aspect of the device, thereby forming a testing sample feature set. The operations further comprise determining a defect associated with the device by applying a defect detection logic to the testing sample feature set, the defect detection logic corresponding to the device. The operations further comprise causing an indication to be provided regarding the defect associated with the device.

[0188] Advantageously, these and other embodiments, as described herein, provide a more robust and efficient approach to defect detection. For example, by analyzing features across consecutive video frames, some of these embodiments of the defect detection system can detect subtle defects that may not be apparent in static imaging conditions. This dynamic approach reveals defects that may only be visible under specific lighting or motion scenarios, significantly improving the accuracy and reliability of the inspection process. Furthermore,these system's ability to adapt and learn from prior detections enhances its efficiency over time, making some of these embodiments particularly effective in identifying emerging or evolving defect patterns.

[0189] In any combination of the above embodiments, determining the defect associated with the device comprises determining a likelihood that the device has an existing defect or will develop a defect during its expected lifetime.

[0190] In any combination of the above embodiments, the indication regarding the defect comprises at least one of a likelihood of the defect, a type of defect, and manufacturing information regarding the device.

[0191] In any combination of the above embodiments, the video segment is generated by capturing, using a camera, image data of the device while the controlling a first motion of the device with respect to the camera or of the camera with respect to the device.

[0192] In any combination of the above embodiments, the first motion comprises a rotational motion, linear motion, tilt motion, or planar motion.

[0193] In any combination of the above embodiments, the first motion is determined based on a type of the device.

[0194] In any combination of the above embodiments, the operations further comprise cause the device determined to have a defect to be segregated for disposal, further testing, or repair.

[0195] In any combination of the above embodiments, the operations further comprise determine a type of defect that characterizes the determined defect; and update the device detection logic to have increased sensitivity for detecting the type of defect.

[0196] In any combination of the above embodiments, the device detection logic is updated by performing at least one of: adding a region of interest (ROI) for determining the testing sample feature, the ROI including a location on the device corresponding to the determined defect; updating an existing ROI for determining the testing sample feature, the existing ROI including the location on the device corresponding to the determined defect, the existing ROI updated by adding or adjusting a weighting associated with the testing sample feature determined from the ROI; updating a coefficient of a defect detection model that is included in the defect detection logic, the coefficient corresponding to a data feature associated with the determined defect; updating a defect detection model that is included in the defect detection logic by fine-tuning the model based on the determined defect; and adjusting a similarity threshold of the defect detection logic.

[0197] In any combination of the above embodiments, processing the video segment to determine the at least one testing sample feature comprises using at least one of: optical flow, an object tracking model, a higher-order motion model, a three-dimensional Convolutional Neural Network (3D CNN), a recurrent neural network (RNN), and a video transformer.

[0198] In any combination of the above embodiments, processing the video segment to determine the at least one testing sample feature uses an RNN that is a Convolutional Long Short-Term Memory, or uses 3D CNN that is an Inflated 3D ConvNet.

[0199] In any combination of the above embodiments, optical flow is utilized by preprocessing at least a portion of the plurality of frames and computing a motion vector between at least two frames of the plurality of frames; the object tracking model is utilized by determining a region of interest in a first frame of the plurality of frames, and tracking the region of interest across multiple frames of the plurality of frames; or the 3D CNN, RNN, or video transformer comprises a model that is trained to determine at least one spatio-temporal feature associated with a defect, the training performed using labeled video data representing a defective device and a non-defective device.

[0200] In any combination of the above embodiments the at least one testing sample feature comprises a first spatio-temporal feature associated with the defect.

[0201] In any combination of the above embodiments, the defect detection logic comprises a machine learning model trained using a defect signature corresponding to the device, the defect signature comprising one or more spatio-temporal features associated with the device defect.

[0202] In any combination of the above embodiments, determining the defect associated with the device comprises determining a similarity of the testing sample feature set and the defect signature, the similarity determined based on a binary-match decision using the model or based on a feature similarity distance, between the testing sample feature set and the defect signature, satisfying a similarity threshold.

[0203] In some embodiments, a computer-implemented method for detecting or predicting defects in a device is provided. The method comprises generating, using at least one camera, video data comprising a plurality of video segments, each video segment having an associated label and comprising a plurality of frames, the plurality of video segments including at least a first segment generated by capturing video of a defective instance of the device having a particular defect and having associated therewith a first segment label indicating the particular defect, and a second segment generated by capturing video of a non-defective instance of thedevice and having a second segment label indicating no defect, the first and second videos of the segments captured under a controlled condition including a first controlled motion of the defective and non-defective instances of the device with respect to the camera, or of the camera with respect to the defective and non-defective instances of the device. The method further comprises training a defect detection model to detect a defect feature corresponding to the particular defect by using the plurality of video segments and their associated labels. The method further comprises generating, from the trained defect detection model, defect detection logic for detecting the particular defect associated with the device. The method further comprises generating machine-readable instructions regarding the controlled condition used for capturing the video and specifying the first controlled motion. The method further comprises generating a device profile corresponding to the device by associating an indication of the device with the defect detection logic and the machine-readable instructions. The method further comprises providing the device profile to be accessible for testing a candidate instance of the device for the particular defect.

[0204] Advantageously, these and other embodiments, as described herein, provide a highly specialized and accurate defect detection systems with functionality enabling tailoring to specific device types. In particular, by generating device profiles that include both defect detection logic and instructions for reproducing controlled testing conditions, some embodiments of the method ensure consistency and reliability in the defect detection process across different testing environments. This approach significantly reduces the likelihood of false positives or missed defects, leading to improved quality control and potentially reducing manufacturing costs associated with defective products. Furthermore, those embodiments using the machine learning functionality are capable of providing improvement of the defect detection capabilities as more data or new data becomes available, thereby making these embodiments increasingly effective over time.

[0205] In any combination of the above embodiments, the method further comprises determining, based on the device profile, that the candidate instance of the device has the particular defect.

[0206] In any combination of the above embodiments, determining that the candidate instance of the device has the particular defect comprises: accessing a video segment representing at least a portion of the candidate instance of the device and comprising a plurality of frames; processing the video segment to determine, using the plurality of frames, at least one testing sample feature representing, across the plurality of frames, an aspect of the device,thereby forming a testing sample feature set; determining the defect associated with the device by applying the defect detection logic to the testing sample feature set; and providing a notification indicating that the candidate instance of the device likely has the particular defect.

[0207] In any combination of the above embodiments, processing the video segment to determine the at least one testing sample feature comprises using at least one of: optical flow, an object tracking model, a three-dimensional Convolutional Neural Network (3D CNN), a recurrent neural network (RNN), and a video transformer.

[0208] In any combination of the above embodiments, the defect detection logic comprises: a defect signature that includes the trained defect detection model; and machine-readable instructions for applying the trained defect detection model to a testing sample feature set corresponding to the candidate instance of the device.

[0209] In any combination of the above embodiments, the defect detection logic comprises: a defect signature that includes one or more spatio-temporal features corresponding to the particular defect and determined from the trained defect detection model; and machine-readable instructions for computing a similarity of the defect signature to a testing sample feature set corresponding to the candidate instance of the device.

[0210] In any combination of the above embodiments, the plurality of video segments further includes a first plurality of video segments ("defective device training data") including the first segment, and a second plurality of video segments ("non-defective device training data") including the second segment.

[0211] In any combination of the above embodiments each video segment of the first plurality of video segments is generated by capturing video of a different defective instance of the device having the particular defect and having associated therewith a corresponding label indicating the particular defect.

[0212] In any combination of the above embodiments each video segment of the second plurality of video segments is generated by capturing video of a different non-defective instance of the device and having associated therewith a corresponding label indicating no defect.

[0213] In any combination of the above embodiments, the defect detection model comprises a three-dimensional Convolutional Neural Network (3D CNN), a Convolutional Long Short-Term Memory (ConvLSTM), or a video transformer.

[0214] In any combination of the above embodiments, the defect detection model comprises a Siamese 3D CNN.

[0215] In any combination of the above embodiments training the defect detection model is performed using positive pairs of video segments from the first plurality and the second plurality, and negative pairs of video segments comprising a video segment from the first plurality of video segments and a video segment from the second plurality of video segments.

[0216] In any combination of the above embodiments, the defect detection model comprises a Siamese 3D CNN.

[0217] In any combination of the above embodiments training the defect detection model is performed using triplet loss by providing a plurality of three-video-segment inputs to the defect detection model, each three-video-segment input comprising either a first combination of two video segment inputs from the defective device training data and a third video segment input from the non-defective device training data, or comprising a second combination of two video segment inputs from the non-defective device training data and a third video segment input from the defective device training data.

[0218] In any combination of the above embodiments, for each combination, the first video segment of the two video segment inputs comprises an anchor example, the second video segment of the two video segment inputs comprises a positive example, and the third video segment input comprises a negative example.

[0219] In any combination of the above embodiments, the device profile includes an indication of the particular defect corresponding to the defect detection logic.

[0220] In any combination of the above embodiments, the machine-readable instructions regarding the controlled condition comprise instructions for controlling a device manipulator to move the candidate instance of the device according to the first controlled motion.

[0221] In any combination of the above embodiments, the controlled condition includes a first lighting condition of the defective and non-defective instances of the device.

[0222] In any combination of the above embodiments the machine-readable instructions regarding the controlled condition comprise instructions for controlling a testing environment lighting source, to reproduce the first lighting condition on the candidate instance of the device during the testing.

[0223] In any combination of the above embodiments, the video data is generated according to a configuration of the at least one camera.

[0224] In any combination of the above embodiments the machine-readable instructions regarding the controlled condition include instructions for operating at least one testing environment camera according to the configuration of the at least one camera.

[0225] In any combination of the above embodiments, the device profile is provided to be accessible for testing via a digital marker on the candidate instance of the device, or on product packaging or labeling associated with the candidate instance of the device.

[0226] In any combination of the above embodiments, the digital marker comprises a quick reference code (QR Code).

[0227] In any combination of the above embodiments determining, based on the device profde, that the candidate instance of the device has the particular defect includes accessing the device profile via scanning the QR code using the at least one camera.Example Computing Environments

[0228] Having described various implementations, several example computing environments suitable for implementing embodiments of the disclosure are now described, including an example computing device and an example distributed computing environment in FIGS.7 and 8, respectively. With reference to FIG. 7, an example computing device is provided and referred to generally as computing device 700. The computing device 700 is but one example of a suitable computing environment and is not intended to suggest any limitation as to the scope of use or functionality of embodiments of the disclosure, and nor should the computing device 700 be interpreted as having any dependency or requirement relating to any one or combination of components illustrated. Accordingly, computing device 700 comprises any type of computing device capable of performing aspects of the operations described in connection with FIGS. 1 and 2, or methods 400, 500, and 600 in FIGS. 4-6. By way of example and not limitation, computing device 700 may be embodied as a personal computer (PC), a laptop computer, a mobile device, a smartphone, a tablet computer, a smart watch, a wearable computer, a virtual-reality (VR) or augmented-reality (AR) device or headset, a smart display such as a smart television, a handheld scanner device, a workstation, any other suitable computer device, or any combination of these delineated devices.

[0229] Embodiments of the disclosure are described in the general context of computer code or machine-useable instructions, including computer-useable or computer-executable instructions, such as program modules, being executed by a computer or other machine suchas a smartphone, a tablet PC, or other mobile device, server, or client device. Generally, program modules, including routines, programs, objects, components, data structures, and the like, refer to code that performs particular tasks or implements particular abstract data types. Embodiments of the disclosure are practiced in a variety of system configurations, including mobile devices, consumer electronics, general-purpose computers, more specialty computing devices, or the like. Embodiments of the disclosure can also be practiced in distributed computing environments where tasks are performed by remote-processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.

[0230] Some embodiments comprise an end-to-end software-based system that operates within system components described herein to operate computer hardware, which may include computerized equipment, to provide system functionality. At a low level, hardware processors generally execute instructions selected from a machine language (also referred to as machine code or native) instruction set for a given processor. The processor recognizes the native instructions and performs corresponding low-level functions related to, for example, logic, control, and memory operations. Low-level software written in machine code can provide more complex functionality to higher-level software. Accordingly, in some embodiments, computer-executable instructions include any software, including low-level software written in machine code, higher-level software such as application software, and any combination thereof. In this regard, the system components can manage resources and provide services for system functionality. Any other variations and combinations thereof are contemplated with the embodiments of the present disclosure.

[0231] With reference to FIG. 7, computing device 700 includes a bus 710 that directly or indirectly couples the following devices: memory 712, one or more processors 714, one or more presentation components 716, one or more input / output (I / O) ports 718, one or more I / O components 720, and an illustrative power supply 722. In one example, bus 710 represents one or more buses (such as an address bus, data bus, or combination thereof). Although the various blocks of FIG. 7 are shown with lines for the sake of clarity, in reality, these blocks represent logical, not necessarily actual, components. For example, a presentation component includes a display device, such as an I / O component. Also, processors have memory. It is recognized that such is the nature of the art and reiterate that the diagram of FIG. 7 is merely illustrative of an exemplary computing device that can be used in connection with one or more embodiments ofthe present disclosure. Distinction is not made between such categories as “workstation,” “server,” “laptop,” or “handheld device,” as all are contemplated within the scope of FIG. 7 and with reference to “computing device.”

[0232] Computing device 700 typically includes a variety of computer- readable media. Computer-readable media can be any available media that can be accessed by computing device 700 and includes both volatile and nonvolatile, removable and non-removable media. By way of example, and not limitation, computer-readable media comprises computer storage media and communication media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, random-access memory (RAM), read-only memory (ROM), electrically erasable programmable readonly memory (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disks (DVDs) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and which can be accessed by computing device 700. Computer storage media does not comprise signals per se. Communication media typically embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner so as to encode information in the signal. By way of example, and not limitation, communication media includes wired media, such as a wired network or direct-wired connection, and wireless media, such as acoustic, radio frequency (RF), infrared, and other wireless media. Combinations of any of the above should also be included within the scope of computer-readable media.

[0233] Memory 712 includes computer storage media in the form of volatile and / or non-volatile memory. In one example, the memory is removable, non-removable, or a combination thereof. Hardware devices include, for example, solid-state memory, hard drives, and optical-disc drives. Computing device 700 includes one or more processors 714 that read data from various entities such as memory 712 or I / O components 720. As used herein and in one example, the term processor or “a processer” refers to more than one computer processor. For example, the term processor (or “a processor”) refers to at least one processor, which may be a physical or virtual processor, such as a computer processor on a virtual machine. The termprocessor (or “a processor”) also may refer to a plurality of processors, each of which may be physical or virtual, such as a multiprocessor system, distributed processing or distributed computing architecture, cloud computing system, or parallel processing by more than a single processor. Further, various operations described herein as being executed or performed by a processor are performed by more than one processor.

[0234] Presentation component(s) 716 presents data indications to a user or other device. Presentation components include, for example, a display device, speaker, printing component, vibrating component, and the like.

[0235] The I / O ports 718 allow computing device 700 to be logically coupled to other devices, including I / O components 720, some of which are built-in. Illustrative components include a microphone, joystick, game pad, satellite dish, scanner, printer, or a wireless device. The I / O components 720 can provide a natural user interface (NUI) that processes air gestures, voice, or other physiological inputs generated by a user. In some instances, inputs are transmitted to an appropriate network element for further processing. An NUI may implement any combination of speech recognition, touch and stylus recognition, facial recognition, biometric recognition, gesture recognition both on screen and adjacent to the screen, air gestures, head and eye tracking, and touch recognition associated with displays on the computing device 700. In one example, the computing device 700 is equipped with depth cameras, such as stereoscopic camera systems, infrared camera systems, red-green-blue (RGB) camera systems, and combinations of these, for gesture detection and recognition. Additionally, the computing device 700 may be equipped with accelerometers or gyroscopes that enable detection of motion. The output of the accelerometers or gyroscopes may be provided to the display of the computing device 700 to render immersive augmented reality or virtual reality.

[0236] Some embodiments of computing device 700 include one or more radio(s) 724 (or similar wireless communication components). The radio transmits and receives radio or wireless communications. Example computing device 700 is a wireless terminal adapted to receive communications and media over various wireless networks. Computing device 700 may communicate via wireless protocols, such as code-division multiple access (“CDMA”), Global System for Mobile (“GSM”) communication, or time-division multiple access (“TDMA”), as well as others, to communicate with other devices. In one embodiment, the radio communication is a short-range connection, a long-range connection, or a combination of both a short-range and a long-range wireless telecommunications connection. In various embodiments, references to “short” and “long” types of connections do not refer to the spatialrelation between two devices. Instead, in general references short range and long range as different categories, or types, of connections (for example, a primary connection and a secondary connection). A short-range connection includes, by way of example and not limitation, a Wi-Fi® connection to a device (for example, mobile hotspot) that provides access to a wireless communications network, such as a wireless local area network (WLAN) connection using the 802.11 protocol; a Bluetooth connection to another computing device is a second example of a short-range connection, or a near-held communication connection. A long-range connection may include a connection using, by way of example and not limitation, one or more of Code-Division Multiple Access (CDMA), General Packet Radio Service (GPRS), Global System for Mobile Communication (GSM), Time-Division Multiple Access (TDMA), and 802.16 protocols.

[0237] Referring now to FIG. 8, an example distributed computing system 800 is illustratively provided, in which implementations of the present disclosure can be employed. In particular, FIG. 8 shows a high-level architecture of an example cloud computing platform 810 that can host a technical solution environment, or a portion thereof (for example, a data trustee environment). It should be understood that this and other arrangements described herein are set forth only as examples. For example, as described above, many of the elements described herein arc implemented as discrete or distributed components or in conjunction with other components, and in any suitable combination and location. Other arrangements and elements (for example, machines, interfaces, functions, orders, and groupings of functions) can be used in addition to or instead of those shown.

[0238] Data centers can support distributed computing system 800 that includes cloud computing platform 810, rack 820, and node 830 (for example, computing devices, processing units, or blades) in rack 820. The technical solution environment can be implemented with cloud computing platform 810, which runs cloud services across different data centers and geographic regions. Cloud computing platform 810 can implement the fabric controller 840 component for provisioning and managing resource allocation, deployment, upgrade, and management of cloud services. Typically, cloud computing platform 810 acts to store data or run service applications in a distributed manner. Cloud computing platform 810 in a data center can be configured to host and support operation of endpoints of a particular service application. In one example, the cloud computing platform 810 is a public cloud, a private cloud, or a dedicated cloud.

[0239] Node 830 can be provisioned with host 850 (for example, operating system or runtime environment) running a defined software stack on node 830. Node 830 can also be configured to perform specialized functionality (for example, computer nodes or storage nodes) within cloud computing platform 810. Node 830 is allocated to run one or more portions of a service application of a tenant. A tenant can refer to a customer utilizing resources of cloud computing platform 810. Service application components of cloud computing platform 810 that support a particular tenant can be referred to as a multitenant infrastructure or tenancy. The terms “service application,” “application,” or “service” are used interchangeably with regards to FIG. 8, and broadly refer to any software, or portions of software, that run on top of, or access storage and computing device locations within, a datacenter.

[0240] When more than one separate service application is being supported by nodes 830, certain nodes 830 are partitioned into virtual machines (for example, virtual machine 852 and virtual machine 854). Physical machines can also concurrently run separate service applications. The virtual machines or physical machines can be configured as individualized computing environments that are supported by resources 860 (for example, hardware resources and software resources) in cloud computing platform 810. It is contemplated that resources can be configured for specific service applications. Further, each service application may be divided into functional portions such that each functional portion is able to run on a separate virtual machine. In cloud computing platform 810, multiple servers may be used to run service applications and perform data storage operations in a cluster. In one embodiment, the servers perform data operations independently but exposed as a single device, referred to as a cluster. Each server in the cluster can be implemented as a node.

[0241] In some embodiments, client device 880 is linked to a service application in cloud computing platform 810. Client device 880 may be any type of computing device, such as a computing device 700 described in FIG. 7, and the client device 880 can be configured to issue commands to cloud computing platform 810. In embodiments, client device 880 communicates with service applications through a virtual Internet Protocol (IP) and load balancer or other means that direct communication requests to designated endpoints in cloud computing platform 810. Gertain components of cloud computing platform 810 communicate with each other over a network (not shown), which includes, without limitation, one or more local area networks (LANs) and / or wide area networks (WANs).Additional Structural and Functional Features of Embodiments of Technical Solutions

[0242] Having identified various components utilized herein, it should be understood that any number of components and arrangements may be employed to achieve the desired functionality within the scope of the present disclosure. For example, the components in the embodiments depicted in the figures are shown with lines for the sake of conceptual clarity. Other arrangements of these and other components may also be implemented. For example, although some components are depicted as single components, many of the elements described herein may be implemented as discrete or distributed components or in conjunction with other components, and in any suitable combination and location. Some elements may be omitted altogether. Moreover, various functions described herein as being performed by one or more entities may be carried out by hardware, firmware, and / or software, as described below. For instance, various functions may be carried out by a processor executing instructions stored in memory. As such, other arrangements and elements (for example, machines, interfaces, functions, orders, and groupings of functions) can be used in addition to or instead of those shown.

[0243] Embodiments described in the paragraphs below may be combined with one or more of the specifically described alternatives. In particular, an embodiment that is claimed may contain a reference, in the alternative, to more than one other embodiment. The embodiment that is claimed may specify a further limitation of the subject matter claimed.

[0244] For purposes of this disclosure, the word “including” has the same broad meaning as the word “comprising,” and the word “accessing” comprises “receiving,” “referencing,” or “retrieving.” Furthermore, the word “communicating” has the same broad meaning as the word “receiving” or “transmitting” facilitated by software or hardware-based buses, receivers, or transmitters using communication media described herein. In addition, words such as “a” and “an,” unless otherwise indicated to the contrary, include the plural as well as the singular. Thus, for example, the constraint of “a feature” is satisfied where one or more features are present. Also, the term “or” includes the conjunctive, the disjunctive, and both (for example, “a or b” includes either a or b, as well as a and b).

[0245] As used herein, the terms “application” or “app” may be employed interchangeably to refer to any software -based program, package, or product that is executable via one or more (physical or virtual) computing machines or devices. An application may be any set of software products that, when executed, provide an end user one or morecomputational and / or data services. In some embodiments, an application may refer to a set of applications that may be executed together to provide the one or more computational and / or data services. The applications included in a set of applications may be executed serially, in parallel, or any combination thereof. The execution of multiple applications (comprising a single application) may be interleaved. For example, an application may include a first application and a second application. An execution of the application may include the serial execution of the first and second application or a parallel execution of the first and second applications. In other embodiments, the execution of the first and second application may be interleaved.

[0246] For purposes of a detailed discussion above, embodiments of the present disclosure are described with reference to a computing device or a distributed computing environment; however the computing device and distributed computing environment depicted herein are non-limiting examples. Moreover, the terms computer system and computing system may be used interchangeably herein, such that a computer system is not limited to a single computing device, and nor does a computing system require a plurality of computing devices. Rather, various aspects of the embodiments of this disclosure may be carried out on a single computing device or a plurality of computing devices, as described herein. Additionally, components can be configured for performing novel aspects of embodiments, where the term “configured for’’ can refer to “programmed to” perform particular tasks or implement particular abstract data types using code. Further, while embodiments of the present disclosure may generally refer to the technical solution environment and the schematics described herein, it is understood that the techniques described may be extended to other implementation contexts.

[0247] Many different arrangements of the various components depicted, as well as components not shown, are possible without departing from the scope of the claims below. Embodiments of the present disclosure have been described with the intent to be illustrative rather than restrictive. Alternative embodiments will become apparent to readers of this disclosure after and because of reading it. Alternative means of implementing the aforementioned can be completed without departing from the scope of the claims below. Certain features and subcombinations are of utility and may be employed without reference to other features and subcombinations and are contemplated within the scope of the claims.

Claims

CLAIMS1. A system for detecting or predicting defects in a medical device, the system comprising: an imaging system operable to capture image data; a device manipulator operable to control a motion of the medical device; at least one processor; a system-user interface; and computer memory having computer-readable instructions embodied thereon, that, when executed by the at least one processor, perform operations comprising: receive a device profile corresponding to the medical device, the device profile including instructions for controlling testing conditions for generating a testing sample corresponding to the medical device; based on the device profile, programmatically control the device manipulator to cause a first motion of the medical device; cause the imaging system to generate the testing sample comprising a video segment representing at least a portion of the medical device, the video segment generated by capturing image data of the medical device during the first motion, the video segment comprising a plurality of frames; processes the video segment to determine, using the plurality of frames, at least one testing sample feature representing, across the plurality of frames, an aspect of the medical device, thereby forming a testing sample feature set; based on the testing sample feature set, determine a likelihood of a defect associated with the medical device; and based on the determined likelihood, cause a notification, regarding the defect associated with the medical device, to be provided via the system-user interface.

2. The system of claim 1, wherein processing the video segment to determine the at least one testing sample feature comprises using at least one of: optical flow, an object tracking model, a three-dimensional Convolutional Neural Network (3D CNN), a recurrent neural network (RNN), and a video transformer.

3. The system of claim 2: wherein optical flow is utilized by preprocessing at least a portion of the plurality of frames and computing a motion vector between at least two frames of the plurality of frames; wherein the object tracking model is utilized by determining a region of interest in a first frame of the plurality of frames, and tracking the region of interest across multiple frames of the plurality of frames; wherein the 3D CNN, RNN, or video transformer comprises a model that is trained to determine at least one spatio-temporal feature associated with a particular defect, the training performed using labeled video data representing a defective medical device, the labeled video data generated by capturing image data of the defective medical device under a controlled condition that is similar to the testing condition, including at least a controlled motion similar to the first motion; or wherein processing the video segment to determine the at least one testing sample feature uses an RNN that is a Convolutional Long Short-Term Memory, or uses 3D CNN that is an Inflated 3D ConvNet.

4. The system of claim 1: wherein the device profile includes a defect detection logic for detecting or predicting defects in the medical device, and wherein the likelihood of the defect associated with the medical device is determined by applying the defect detection logic to the testing sample feature set; and wherein the defect detection logic comprises a machine learning model trained using a defect signature corresponding to the medical device.

5. The system of claim 4: wherein the likelihood of the defect associated with the medical device represents a similarity score corresponding to the similarity of the testing sample feature set and the defect signature; wherein the testing sample feature set comprises at least a portion of the video segment and the defect signature comprises video data representing at least one instance of a particular defect occurring in a defective version of the medical device; wherein applying the defect detection logic to the testing sample feature set comprises determining a similarity of the testing sample feature set to a defect signature corresponding to the medical device, the similarity determined based on a binary-match decision or a feature similarity distance, of the testing sample feature set and the defect signature, satisfying a similarity threshold, the defect signature comprising one or more spatiotemporal features associated with defects; and wherein the defect signature is determined from a model trained using video data representing a defective medical device and a non-defective medical device, the video data generated by capturing image data of the defective medical device and the non-defective medical device under a controlled condition that is similar to the testing condition, including at least a controlled motion similar to the first motion.

6. The system of claim 1, wherein the notification regarding the defect includes an indication of the determined likelihood of the defect, an indication of a defect type, and manufacturing information regarding the medical device.

7. The system of claim 1 : wherein the instructions of the device profile for controlling testing conditions include instructions for controlling the imaging system to generate the testing sample; the imaging system is caused to generate the testing sample based on the instructions for controlling the imaging system; and wherein imaging system includes a camera and wherein the instructions for controlling the imaging system comprise instructions specifying at least one of a frame rate for the camera, a shutter speed, a minimum or a maximum number of frames to capture for the video segment, and a duration of video to capture for the video segment.

8. The system of claim 1 further comprising a lighting system, wherein the instructions of the device profile for controlling testing conditions include instructions for controlling the lighting system, and wherein the instructions for controlling the lighting system comprise instructions specifying at least one of a lighting color temperature, lighting strobe frequency, lumen level, and angle of incidence with regard to the medical device.

9. The system of claim 1, wherein: the first motion comprises a rotational motion, a linear motion, a tilt motion, or a planar motion; and the operations further comprise programmatically controlling the device manipulator to cause a second motion of the medical device, the second motion being different than the first motion.

10. A computing system for detecting or predicting defects in a device, the system comprising: at least one processor; and computer memory having computer-readable instructions embodied thereon, that, when executed by the at least one processor, perform operations comprising: access a video segment representing at least a portion of the device and comprising a plurality of frames; processes the video segment to determine, using the plurality of frames, at least one testing sample feature representing, across the plurality of frames, an aspect of the device, thereby forming a testing sample feature set; determine a defect associated with the device by applying a defect detection logic to the testing sample feature set, the defect detection logic corresponding to the device; and cause an indication to be provided regarding the defect associated with the device.

11. The system of claim 10, wherein determining the defect associated with the device comprises determining a likelihood that the device has an existing defect or will develop a defect during its expected lifetime.

12. The system of claim 10, wherein the indication regarding the defect comprises at least one of a likelihood of the defect, a type of defect, and manufacturing information regarding the device, and wherein the operations further comprise cause the device determined to have a defect to be segregated for disposal, further testing, or repair.

13. The system of claim 10, wherein the video segment is generated by capturing, using a camera, image data of the device while controlling a first motion of the device with respect to the camera or of the camera with respect to the device, the first motion comprising a rotational motion, linear motion, tilt motion, or planar motion, and wherein the first motion is determined based on a type of the device.

14. The system of claim 10, wherein the operations further comprise: determine a type of defect that characterizes the determined defect; and update the device detection logic to have increased sensitivity for detecting the type of defect, wherein the device detection logic is updated by performing at least one of: adding a region of interest (ROI) for determining the testing sample feature, the ROI including a location on the device corresponding to the determined defect; updating an existing ROI for determining the testing sample feature, the existing ROI including the location on the device corresponding to the determined defect, the existing ROI updated by adding or adjusting a weighting associated with the testing sample feature determined from the ROI; updating a coefficient of a defect detection model that is included in the defect detection logic, the coefficient corresponding to a data feature associated with the determined defect; updating a defect detection model that is included in the defect detection logic by fine-tuning the model based on the determined defect; or adjusting a similarity threshold of the defect detection logic.

15. A computer-implemented method for detecting or predicting defects in a device, comprising: for the device, generating, using at least one camera, video data comprising a plurality of video segments, each video segment having an associated label and comprising a plurality of frames, the plurality of video segments including at least: a first segment generated by capturing video of a defective instance of the device having a particular defect and having associated therewith a first segment label indicating the particular defect; and a second segment generated by capturing video of a non-defective instance of the device and having a second segment label indicating no defect, wherein the first and second videos of the segments are captured under a controlled condition including a first controlled motion of the defective and non-defective instances of the device with respect to the camera, or of the camera with respect to the defective and non-defective instances of the device; training a defect detection model to detect a defect feature corresponding to the particular defect by using the plurality of video segments and their associated labels; generating, from the trained defect detection model, defect detection logic for detecting the particular defect associated with the device; generating machine-readable instructions regarding the controlled condition used for capturing the video and specifying the first controlled motion; generating a device profde corresponding to the device by associating an indication of the device with the defect detection logic and the machine-readable instructions; and providing the device profile to be accessible for testing a candidate instance of the device for the particular defect.

16. The method of claim 15, further comprising determining, based on the device profile, that the candidate instance of the device has the particular defect.

17. The method of claim 16, wherein determining that the candidate instance of the device has the particular defect comprises: accessing a video segment representing at least a portion of the candidate instance of the device and comprising a plurality of frames; processing the video segment to determine, using the plurality of frames, at least one testing sample feature representing, across the plurality of frames, an aspect of the device, thereby forming a testing sample feature set; determining the defect associated with the device by applying the defect detection logic to the testing sample feature set; providing a notification indicating that the candidate instance of the device likely has the particular defect, and wherein processing the video segment to determine the at least one testing sample feature comprises using at least one of: optical flow, an object tracking model, a three-dimensional Convolutional Neural Network (3D CNN), a recurrent neural network (RNN), and a video transformer.

18. The method of claim 15, wherein the defect detection logic comprises: a defect signature that includes the trained defect detection model, and machine-readable instructions for applying the trained defect detection model to a testing sample feature set corresponding to the candidate instance of the device; or the defect signature that includes one or more spatio-temporal features corresponding to the particular defect and determined from the trained defect detection model; and machine-readable instructions for computing a similarity of the defect signature to a testing sample feature set corresponding to the candidate instance of the device.

19. The method of claim 15, wherein the plurality of video segments further includes a first plurality of video segments (“defective device training data”) including the first segment, and a second plurality of video segments (“non-defective device training data”) including the second segment; wherein each video segment of the first plurality of video segments is generated by capturing video of a different defective instance of the device having the particular defect and having associated therewith a corresponding label indicating the particular defect; and wherein each video segment of the second plurality of video segments is generated by capturing video of a different non-defective instance of the device and having associated therewith a corresponding label indicating no defect.

20. The method of claim 15, wherein the device profile is provided to be accessible for testing via a digital marker on the candidate instance of the device, or on product packaging or labeling associated with the candidate instance of the device.

Images