Machine learning based visual maintenance inspection system

Abstract

A system for condition monitoring and predictive maintenance of in-motion components within an industrial system is disclosed. A centralized controller receives image data from vision sensors and operational data from telemetry sensors. An indexing module detects each component at a trigger location, assigns a sequential index during a full system revolution, and associates acquired data with corresponding subcomponents. A measurement module analyzes images to determine characteristics such as wear, deformation, or misalignment, while a predictive model evaluates the measurement data and historical maintenance information to estimate future wear and replacement intervals. An alarm manager compares the measurements to threshold values and generates multi-level condition alarms. The system produces a consolidated report identifying component-specific wear conditions, predicted maintenance timing, and recommended actions. This architecture enables proactive maintenance, reduces downtime, and improves operational reliability in large-scale automated systems by providing continuous or scheduled monitoring and automated assessment of subcomponent health.

Description

MACHINE LEARNING BASED VISUAL MAINTENANCE INSPECTION SYSTEMBACKGROUND OF THE INVENTION

[0001] Automated material handling systems are critical in various industries, including manufacturing, warehousing, and logistics. These systems rely on conveyor belts and sorters to move materials efficiently. Regular maintenance is essential to ensure continuous and reliable operation. Traditional maintenance methods are usually reactive, responding to failures after they occur, or are based on scheduled intervals, which may not accurately reflect the equipment's condition. Recent advancements in machine learning and sensor technologies offer opportunities to enhance maintenance practices by providing real-time condition monitoring and predictive insights. However, there is a need for an integrated system that leverages these technologies to optimize maintenance operations in automated material handling systems.SUMMARY OF THE INVENTION

[0002] This invention provides a system and methodology for machine learning-based visual maintenance inspection. The system enhances the efficiency and reliability of automated material handling systems, particularly conveyor systems and sorters. This system utilizes a combination of one or more key technologies: vision sensors to capture visual data, telemetry sensors to monitor the performance of individual components, and computational algorithms to analyze this data. Specifically, the system works by intelligently matching subcomponents, parts of a larger unit, to their corresponding parent components. This matching process allows for the collection of valuable data points, including critical metrics such as power consumption, mechanical wear, alignment issues, and other relevant factors. The collected data enables the creation of alerts per component and subcomponent, facilitating predictive maintenance, and ultimately, generating tailored corrective and preventative maintenance tasks.FIELD OF THE INVENTION

[0003] This invention pertains to automated material handling systems, particularly those comprised of a complex network of interconnected subcomponents, components that, without intelligent monitoring, would require significant manual inspection. These systems are frequently utilized in large scale operations such as conveyor systems, sorters, and larger logistics hubs. Currently, manual inspection of each individual subcomponent is time consuming, labor intensive, and prone to human error.BRIEF DESCRIPTION OF THE INVENTION

[0004] Figure 1 provides a layered representation of the system according to one embodiment, wherein every subcomponent maps to a component, and a collection of these components, combined with additional standalone elements, makes up the full system.

[0005] Figure 2 is a flow diagram that explains the full data collection process and report for independent components and for an array of components in the system.

[0006] Figure 3 is a connection diagram focused on Ethernet links with communication paths between the various devices involved in the invention.

[0007] Figure 4 illustrates an example of a two-arm carbon brush designed to drain static energy from a moving component according to one embodiment.

[0008] Figure 5 shows an example of one of the carbon brushes and a defect of interest: wear.

[0009] Figure 6 shows an example of one of the carbon brushes and the defect of interest: position, spring tension.

[0010] Figure 7 illustrates another example application of the system with a diagram showing wheels from the side, wherein each one of the wheels portrayed has a different condition: good state (convex), wear wheel (concave), axis bent, and wear wheel flat.

[0011] Figure 8 illustrates an example of a bent component application (in this case a screw is used as example).

[0012] Figure 9 is a flow diagram of an alarm manager internal process to output the report of each item according to one embodiment.DETAILED DESCRIPTION OF THE INVENTION

[0013] The architecture of the system of the present invention shown in the Figures combines a centralized computing system (IPC) 300 with hardware components for data collection. The system operates as a localized solution, connecting devices such as vision sensors (303, 304, 305), programmable logic controllers (PLCs) 306, variable frequency drives (VFDs) (309, 311), encoders (308, 310), and other telemetry sensors to monitor various aspects of a system, such as a package sorting system, and provide one or more reports relating to the status of each aspect, component, or subcomponent, including when a system component is worn and needs replacement, or when a system component may need future replacement based on prediction.

[0014] In one embodiment, the invention provides comprehensive condition monitoring and predictive maintenance for equipment within a defined system. The central computing system 300 may focus on input received from computer vision tools and other sensors, and data processing and integration of the input received with predictive modeling modules, along with Al pattern recognition to intelligently provide a report of the current state and maintenance recommendations of the system components. Depending on the data type and gathering data process, the analysis could be running with live data or scheduled to all or certain tasks.

[0015] In one embodiment, the invention implements a subcomponent to in-motion component matching algorithm. A diagram of one such system is depicted in Fig. 1. In this example, the system 100 depicted is a sorting system 100, including a series of components 101 that may be carriages, carriers, trays or the like. In such a system 100, these components 101 may be moving or “in-motion” components 101 , that travel through a facility along a rail, or via a conveyor or another device, and each of the components 101 may include one or more subcomponents for transporting the components 101. In the illustrated embodiment, these subcomponents includefasteners 102, 103, wheels 104, 105, and gear 106. For example, in the case where the components are carriages 101 , each carriage 101 may include such subcomponents for supporting or moving the carriages 101 and each of the subcomponents may therefore wear over time. In one particular embodiment, each carriage 101 may be supported on one or more wheels 104, and driven by one or more gears 106, and the wheels 104 and / or gears 106 may be attached to the carriage 101 or a frame member (not shown) with one or more fasteners 102, which are depicted as screw or nail-like fasteners but may otherwise be nut and bolt style fasteners or another fastener type.

[0016] In one embodiment, the system 100 may automatically associate the various subcomponents (102, 103, 104, 105, 106) with one of the in-motion components 101 such that the system can uniquely identify each of the in-motion components 101 and its associated subcomponents (102, 103, 104, 105) within a wide-scale system, which may be particularly crucial for large scale systems 100, which may include one or more of the systems 100, each with hundreds of in-motion components 101 and subcomponents (102, 103, 104, 105), as the system may initiate a comprehensive data collection process of each individual inspection item, wherein each individual inspection item, such as the vision tools and other sensors, may be mapped to a location in one or more of the systems 100. The algorithm leverages a complete inspection of all components 101 and subcomponents and is better described on Figure 2.

[0017] The flow diagram on Figure 2 shows the process of obtaining data for each active subcomponent within a system 100, analyzing the data to determine whether a particular subcomponent is determined to be “worn” or predicting, based on the measured data, when the subcomponent will be worn and need replacement, outputting the measured data and the prediction to an alarm manager that can determine the type of output to provide to the user, and generating a report with the desired output. The process shown in Figure 2 will iterate depending on the total number of components in the system. Once data is obtained for every component, including subcomponents, then a full data sample has been collected.

[0018] In order to collect this data correctly the system 100 begins indexing components 101 past a camera or other sensor 202, questioning 201 whether each component 101 has subcomponents that are active. If a subcomponent is recognized by the camera sensor 202 then it enters an indexing loop by passing a trigger sensor 206 at a known initial location. The first component 101 to pass the trigger sensor becomes component “1” and further components are successively indexed until a full revolution has been made and the total number of components 101 and associated active subcomponents have passed the trigger sensor 206. As the system 101 then continues to index loop 210 the components 101, it will trigger 209 an image acquisition 205 of each successive component. For each indexed component, if a complete revolution has not been made, the system 100 registers an invalid sample 213. The system 100 will continue to index and gather images of each successive component and subcomponent until images of allactive subcomponents have been gathered. Once a full revolution has been made and all components have been indexed 211, the system 100 registers that a valid sample 214 has been taken.

[0019] While on the loop, the camera sensors will be taking images 205 of the subcomponent in question for each in-motion component trigger 209, which also increased the component index by one. Once this loop is finished and the index matches the number of components 211, then the valid sample 214 can be processed 217 by the necessary vision tools to generate the measurement data 212 relating to one or more characteristics of the imaged subcomponents.

[0020] In the event that the component to measure / test does not have multiple active subcomponents 201 to inspect or there is no camera 202 installed for analysis of that subcomponent 201 , then a power on self-test cycle 203 may be conducted, in which a component (e.g. motor on conveyor, motor on sorter, etc) is activated to run a predetermined sequence in order to evaluate its functionality and efficiency 207. Independently of extra components to be analyzed by cameras or not, some components may have other test sequences 204, and these sequences may be activated via a test run 208, and just like the subcomponent measurement data, the results 212 of these test runs may be input into a prediction model 215 to later run the alarm manager 216 and get a full report 218 with all data, results, alarms, predictions and recommendations. This process or running a self-test 203 and any other available test, may be closed 219 and repeated for every component.

[0021] As the components 101 are indexed past one or more cameras (303, 304, 305), which are described in more detail below, the cameras (303, 304, 305) automatically image each subcomponent and match the image data for each subcomponent with the sequential index location of the imaged component so that the image data of each component 101 and its subcomponents is mapped to the location of that particular component (i.e., in a system of 100 total components, the system 100 knows that image data for a component is n / 100 depending on the sequence order of the component 101). This automated process generates a database of subcomponent relationships 212 with its pertaining in-motion component 101 , for easy tracking and location when a specified in-motion component 101 needs to be replaced or maintained. The results are compared with a historical maintenance log, allowing for the creation of a predictive maintenance model 215. The model analyzes past failures, failure rates, and component degradation patterns to generate recommendations for proactive maintenance tasks, linked to an SOP library to show possible routes of action, minimizing downtime and also optimizing resource allocation for critical spare units on the system.

[0022] In addition to the prediction model 215, the system 100 includes an alarm manager 216 for comparing the measured data to thresholds and presenting one or more alarms based on the comparison. An alarm manager flow diagram is shown on Figure 9, and portrays a 3 level alarm process to highlight the components 101 health. In this example, starting at 900 with theinspection of a new measured data sample 901, the alarm manager 216 may first log into the data history 902 for the component 101 and then, regardless of the preceding data 903, the alarm manager may assess the measured data for a component 101 (or its subcomponents) and compare to known thresholds for various wear states of such a component 904 to determine a particular alarm (or no alarm) for the component. If the data passes the threshold, then the alarm manager may generate a report or otherwise signal that the component or subcomponent is “good” 909. Otherwise, if the measured data does not meet the threshold, it may fall into one of three alarm levels, including a “Level 3” 905, “Level 2” 906, and “Level 1” 907, depending on the comparison of the measurement to the threshold, and as a result present the respective alarm (910, 911, 912) and then create an alarm manager report 908 and finish 913 with submitting that to a final report 218 for the system 100 along with the prediction model output 215.

[0023] Vision sensors, such as cameras (303, 304, 305), are strategically placed in the system 100 to inspect components or subcomponents of interest as they move along the system, capturing images of such items, this way inspection time can be reduced, delaying but not excluding manual inspection. These cameras (303, 304, 305) may be positioned at key inspection points along the system flow path in locations where the cameras (or other measurement sensors) provide clearance for the moving components 101 yet are still able to clearly see and take images and measurements of the features of the component or subcomponent to be analyzed. With reference to Fig. 3, exemplary electrical connections are shown between the various system measurement and control components, including the cameras (303, 304, 305), programmable logic controllers (PLCs) 306, variable frequency drives (VFDs) (309, 311), encoders (308, 310), controllers 300, 301 and other telemetry sensors. The VFDs (309,311) may be connected to conveyors or sorters to drive motion at variable speeds. The encoders (308, 310) serve as feedback devices that measure the motion of rotating or moving components 101 and report that information to the controller 300. The encoders (308, 310) function is to provide precise, real-time data about position, speed, direction, and sometimes acceleration of a motor shaft, conveyor roller, or other mechanical element.

[0024] The cameras or vision sensors (303, 304, 305) in active use for each component 101 inspected may be specifically configured to measure and detect surface wear, misalignment, cracks, deformation, corrosion, bent or loose fasteners, abnormal wheel profiles, and other visible anomalies that indicate developing mechanical issues. Complementing the vision sensors (303, 304, 305) may be a variety of telemetry sensors that can be tied to the system in order to provide quantitative condition monitoring data, such as vibration measuring, including measuring the frequency and amplitude of mechanical vibrations to reveal possible mechanical faults.

[0025] The present invention may be designed to contain the configurations needed for each module forming a jobset of this system, which will include but not limited to the vision tools occupied, the available tests, thresholds and settings for each element, or the machine learningmodel or prediction model to properly update with newly acquired operational data to improve prediction accuracy and diagnostic capability.

[0026] A few examples of the use of this invention are provided below, starting with one example of a component that may be measured for wear by the system, in this case a carbon brush, shown in Figure 4. The carbon brush may consist of a two-sided arm 400 (or a single-sided arm 401 or 402) that holds the carbon brush. In one embodiment, a carbon brush may be used in this system for providing electricity to moving components such as carriages, belts or other components 101. Relevant inspection measurements for a carbon brush may include wear 500 and arm position, which Figure 6 depicts as initial position 600 in light lines and over-time position or incorrect position 601 in solid lines since the arms move up and down depending on spring tension within the arms 401, 402). In this example, the cameras (303, 304, 305) would be positioned to capture images of the position of the carbon brush arms 401 , 402, and therefore deliver image data to the controller 300 relating to the arm position that could be analyzed using the predictive metrics to determine a projected wear date and could further be compared to threshold data in the alarm manager 215 to determine an appropriate alarm to present.

[0027] A second example involves the identification of bent components, exemplified in Figure 8. In this figure, a bent screw 800 is shown alongside a non-bent screw 801 (which is representative of various types of fasteners such as screws and bolts). These components may be part of moving subcomponents within the system 100 or may remain stationary, and their proper orientation and integrity can be automatically detected by the one or more cameras (303, 304, 305). In this example, the cameras (303, 304, 305) would be positioned to capture images of the position of the fastener 800, 801, and therefore deliver image data to the controller 300 relating to the fastener condition that could be analyzed using the predictive metrics to determine a projected wear date and could further be compared to threshold data in the alarm manager 215 to determine an appropriate alarm to present.

[0028] Another example, shown in Figure 7, is the inspection of wheels in motion systems, which are common in many industries. As long as the wheels are visible, a camera (303, 304, 205) can be placed to capture images, either of a single wheel or as part of a moving component 101. Wheels 700 can be checked from above to see if they are perfectly round and to spot uneven wear, or from the side, as in the diagrams. In these examples: 700 shows a good wheel with a convex profile and no wear, with its diameter measurable; 701 is a concave wheel with visible wear, where the diameter falls below a predetermined acceptable limit; 702 is a bent wheel with the axis at a certain angle; and 703 is a flat wheel that might still work but is showing early signs of failure (i.e., lessening convex shape). This method helps catch issues early and ensures wheels keep working properly, whether they are moving or stationary. In this example, the cameras (303, 304, 305) would be positioned to capture images of the position of the wheel(s), and therefore deliver image data to the controller 300 relating to the wheel shape that could beanalyzed using the predictive metrics to determine a projected wear date and could further be compared to threshold data in the alarm manager 215 to determine an appropriate alarm to present.

[0029] This fully integrated architecture represents a paradigm shift in system management, fundamentally transforming how we approach equipment maintenance. Designed with continuous the possibility of real-time or scheduled monitoring of every asset, allowing to detect potential faults or anomalies that could escalate into costly, unplanned downtime before they significantly impact operations. This proactive approach empowers the maintenance teams to direct resources precisely where and when they are needed, minimizing disruption and maximizing the lifespan of critical equipment along with optimizing spare parts inventory. Ultimately, this tool will dramatically reduce stress on the system, bolstering overall reliability and fostering a sustainable operational environment.

Claims

Claims1. A system for maintenance inspection support and detection of mechanical defects in automated material handling systems, comprising: at least one moving component of the material handling system; at least one motor capable of driving the at least one moving component; vision sensors positioned strategically to capture images of the at least one moving component in the form of image data; at least one telemetry sensor selected from the group including vibration sensors and photoelectric sensors, the at least one telemetry sensor configured to gather telemetry data from the at least one moving component; a data processing unit configured to receive and analyze the image data and the telemetry data; an algorithm to track and match subcomponents of the at least one moving component to their associated component; an alarm manager module configured to compare at least one of the image data and the telemetry data to a threshold and determine, based on the comparison, one of multiple alarms to present; and a prediction module to assess at least one of the image data and the telemetry data and, based on a comparison of the at least one of the image data and telemetry data to known wear pattern data of the at least one component, and provide a report indicative of a predicted wear pattern for the at least one component.

2. The system of claim 1 , wherein the alarm manager notifies maintenance personnel of required maintenance actions through the reports or automated emails.

3. The system of claim 1, wherein the algorithm includes a machine learning algorithm trained on historical data and continuously updated with new data to improve prediction accuracy.

4. The system of claim 1 , wherein the vision sensors are configured to detect surface wear, misalignment, and other visible anomalies in the at least one component.

5. The system of claim 1 , wherein the telemetry data includes vibration data collected by the telemetry sensors.

6. The system of claim 1 , wherein the system generates maintenance schedules and detailed reports to guide maintenance personnel based on the image data and the telemetry data.

7. A system for condition monitoring and predictive maintenance of a plurality of in-motion components within an industrial system, the system comprising: a centralized computing controller configured to receive, store, and process measurement data; one or more vision sensors positioned along a path of travel of the in-motion components and configured to acquire images of subcomponents associated with each of the in-motion components; one or more telemetry sensors configured to obtain non-image operational data from equipment within the industrial system; an indexing module executable by the controller and configured to:(i) detect passage of each in-motion component past a trigger location,(ii) assign a sequential index to each detected in-motion component during a revolution of the industrial system, and(iii) map image data and telemetry data of the subcomponents to the assigned index of the corresponding in-motion component; a measurement module configured to analyze the acquired images to generate quantitative measurement data for one or more characteristics of the subcomponents; a predictive model executed by the controller and configured to determine, based on the measurement data and historical maintenance records, a predicted wear state or predicted replacement time for each subcomponent; and an alarm manager configured to compare the measurement data to one or more thresholds and to generate an alarm level for the in-motion component or the subcomponent based on the comparison; wherein the controller is further configured to generate a report identifying, for each indexed in-motion component, its associated subcomponents, detected wear conditions, predicted maintenance intervals, and corresponding alarm levels.

8. The system of claim 7, wherein the vision sensors comprise cameras arranged at inspection points providing visual access to subcomponents while maintaining clearance from the in-motion components.

9. The system of claim 7, wherein the telemetry sensors comprise vibration sensors, encoders, or programmable logic controller (PLC)-connected sensors providing real-time mechanical or electrical operating parameters.

10. The system of claim 7, wherein the controller is further configured to integrate image- derived measurement data with telemetry data for multi-modal condition assessment.

11. The system of claim 7, wherein the encoders provide position, speed, and direction feedback to correlate motion data with the sequential index assigned to the in-motion components.

12. The system of claim 7, wherein the indexing module registers a valid sample only after all in-motion components complete a full revolution and images of all active subcomponents have been acquired.

13. The system of claim 7, wherein an invalid sample is registered when the full revolution has not been completed before measurement analysis is initiated.

14. The system of claim 7, wherein the mapping of subcomponents to their respective inmotion components establishes a relational database of component-subcomponent associations for maintenance tracking.

15. The system of claim 7, wherein the measurement module is configured to detect surface wear, misalignment, cracks, deformation, corrosion, bent fasteners, wheel profile deviation, or arm position changes in the subcomponents.

16. The system of claim 7, wherein the predictive model employs a machine-learning algorithm trained on historical failures, degradation patterns, and maintenance logs to refine predicted wear dates.

17. The system of claim 7, wherein the predictive model outputs recommended proactive maintenance actions linked to a standardized operating procedure (SOP) library.

18. The system of claim 7, wherein the alarm manager categorizes alarm states into at least three levels, including a first alarm level indicating severe wear, a second alarm level indicating moderate wear, and a third alarm level indicating early wear.

19. The system of claim 7, wherein upon determining that measurement data exceeds a wear threshold, the alarm manager designates the corresponding component as within acceptable limits and generates a non-alarm "good" indication.

20. The system of claim 7, wherein the controller is configured to operate the analysis modules in real-time or on a scheduled basis for selected tasks.

Images