Automatic vision system and method for measuring and inspecting gap and flush in vehicle assembly line

Abstract

An example monocular vision system can emulate multi-view cameras for measuring and inspecting gaps and flushes on vehicle assembly lines. The example system can integrate both vision technology and methods for image processing, dimension estimation, control chart analysis, and processing configuration. A high-resolution camera can include integrated lighting and can capture an image sequence as vehicles move along an inspection line. The processing unit can include advanced modules for image filtering, area localization, edge and point detection, and derive dimension in pixel units for target measure location. Using regression models, the system can convert pixel distances into physical dimensions, leveraging multi-view camera perspectives to enhance accuracy. A graphical user interface can support live streaming, facilitates real-time monitoring, and configures image processing parameters. The system can include a storage unit for archiving processed data and utilizes GPUs and parallel computing to ensure efficiency and speed, enabling real-time monitoring in production.

Description

AUTOMATIC VISION SYSTEM AND METHOD FOR MEASURING AND INSPECTING GAP AND FLUSH IN VEHICLE ASSEMBLY LINECROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U. S. provisional patent application No.63 / 710,855, filed on October 23, 2024, and titled “AUTOMATIC VISION SYSTEM AND METHOD FOR MEASURING AND INSPECTING GAP AND FLUSH IN VEHICLE ASSEMBLY LINE,” the disclosure of which is expressly incorporated herein by reference in its entirety.BACKGROUND

[0002] Dimension inspection is a manufacturing process that is used to identify non-conformities / defects in assembled parts during manufacturing. Automatic dimension inspection systems include camera systems and image processing systems that can image assemblies of parts and determine whether they conform to manufacturing standards.Improvements to dimension inspection systems can improve manufacturing by improving the detection of defects.SUMMARY

[0003] The present disclosure includes a non-contact monocular vision system and associated methods designed for the automated measurement and inspection of gaps and flushes between the car body and its fitted panels. Implementations can include a comprehensive vision system, which integrates hardware and software, and advanced image processing methods extract pixel measurements from images and leverage statistical models to accurately predict gap and flush dimensions, and control chart to monitor the trend and deviations in these dimensions during inspection.

[0004] In some aspects, implementations of the present disclosure include a vision system for measuring and inspecting gaps and flushes of car body on assembly lines, including: a camera configured to capture a sequence of monocular images as vehicles pass along an inspection line; a controller operably coupled to the camera, the controller including a processor and a memory with computer-readable instructions stored thereon, that, when executed by the processor, cause the processor to: receive the sequence of images from the camera; generate a first processed image based on a first image of the sequence of images by localizing a region of interest by template matching and / or reinforcement learning; generate asecond processed image based on a second image of the sequence of images by localizing a region of interest by template matching and / or reinforcement learning; estimate a manufacturing dimension based on the first processed image and the second processed image; and output the estimated manufacturing dimension.

[0005] In some aspects, implementations of the present disclosure include a vision system, wherein generating the first processed image includes performing at least one of: image filtering, area localization, edge detection, point detection, and / or pixel distance computing on a first image of the sequence of images.

[0006] In some aspects, implementations of the present disclosure include a vision system, wherein generating the second processed image includes performing at least one of: image filtering, area localization, edge detection, point detection, and / or pixel distance computing on a second image of the sequence of images.

[0007] In some aspects, implementations of the present disclosure include a vision system, wherein estimating a manufacturing dimension based on the first processed image and the second processed image includes detecting a movement of an object between the first processed image and the second processed image.

[0008] In some aspects, implementations of the present disclosure include a vision system, wherein the manufacturing dimension includes a measurement of a gap between a pair of panels or a measurement of a flush joint between a pair of panels.

[0009] In some aspects, implementations of the present disclosure include a system, wherein the camera includes an integrated light.

[0010] In some aspects, implementations of the present disclosure include a system, wherein the camera is positioned at a predetermined distance from the inspection line and at a predetermined height, with images captured at defined intervals to ensure comprehensive coverage and accurate measurement of gaps and flushes.

[0011] In some aspects, implementations of the present disclosure include a system, wherein integrated lighting is configured to illuminate a target area's edge structure

[0012] In some aspects, implementations of the present disclosure include a system, wherein the memory contains further instructions configured to: execute image filtering to select qualified images based on an appearance, clarity, or movement of a vehicle in the sequence of images, from a last frame of the sequence of images to a current frame of the sequence of images within a predefined window; perform advanced image processing techniques to localize a measure area, extract an edge structure of measure area, determine a measure coordinates of each measure point, and compute a pixel distance between twomeasure points; and implement an outlier handling method to fix missing values and extreme outliers in pixel distances from each image.

[0013] In some aspects, implementations of the present disclosure include a system, further including a GPU, wherein the processor is configured to generate the first processed image and the second processed image by the GPU, and wherein the processing of the first processed image and the second processed image are performed in real time.

[0014] In some aspects, implementations of the present disclosure include a system, wherein the estimated manufacturing dimension is based in part on a regression model configured to convert pixel distances into physical units of gap and flush dimensions.

[0015] In some aspects, implementations of the present disclosure include a system, wherein the system further includes a panel positioning mechanism and the manufacturing dimension includes a gap between a first panel and a second panel, and wherein the memory contains further instructions configured to: control the panel positioning system to optimize the gap between the first panel and the second panel.

[0016] In some aspects, implementations of the present disclosure include a system, further including a GUI configured to support live streaming of the inspection process, provides data search capabilities for root cause analysis, facilitates real-time monitoring of dimensions and control charts, and enables configuration of processing parameters and model tuning.

[0017] In some aspects, implementations of the present disclosure include a method including: capturing images of a manufactured object using a camera as the manufactured object passes along an inspection line; processing the images to derive a plurality of dimensions of the manufactured object; converting the dimensions in pixels to physical measurements by a prediction model for spatial calibration including a second-order polynomial regression; and controlling an assembly line based on the physical measurements or tuning an image processing parameter used to process the image based on the physical measurements.

[0018] In some aspects, implementations of the present disclosure include a method, wherein the step of processing the images includes using a Neural Network framework such as ResNet, CNN, or Yolo to classify images based on vehicle sections and a movement detection method designed to detect movement with a robust move ratio indicator, ensuring that only pertinent images are processed.

[0019] In some aspects, implementations of the present disclosure include a method, wherein step of processing the images further includes: using GPU-accelerated templatematching and / or reinforcement learning to localize a measure area; extracting an edge structure of the measure area by an enhanced Canny edge detection using brightness and improved morphological operations; and detecting a precise measure point coordinates based on proximity to a centroid of the detected measure area.

[0020] In some aspects, implementations of the present disclosure include a method, wherein dimension prediction is performed using a second-order regression model to convert pixel measurements into physical units. This method incorporates a technique that mimics multi-view camera perspectives by categorizing sequential images into different view groups based on measure coordinates, thereby enhancing an accuracy of dimension predictions, wherein a mean and standard deviation of a pixel distance in each group, along with a second-order interaction term, are fed into the regression model as input features.

[0021] In some aspects, implementations of the present disclosure include a method, further including using outlier detection techniques to verify a reliability of pixel distance values and measurements.

[0022] In some aspects, implementations of the present disclosure include a method, wherein controlling the assembly line includes using control charts with a combination of overall Hotelling T2and univariate EWMA control charts for ongoing quality assurance.

[0023] In some aspects, implementations of the present disclosure include a non-transitory computer-readable medium storing having instructions stored therein, wherein execution of the instructions by a processor, causes the processor to: receive the sequence of images from the camera; generate a first processed image based on a first image of the sequence of images by localizing a region of interest by template matching and / or reinforcement learning; generate a second processed image based on a second image of the sequence of images by localizing a region of interest by template matching and / or reinforcement learning; estimate a manufacturing dimension based on the first processed image and the second processed image; and output the estimated manufacturing dimension.

[0024] It should be understood that the above-described subject matter may also be implemented as a computer-controlled apparatus, a computer process, a computing system, or an article of manufacture, such as a computer-readable storage medium.

[0025]

[0026] Other systems, methods, features and / or advantages will be or may become apparent to one with skill in the art upon examination of the following drawings and detailed description. It is intended that all such additional systems, methods, features and / or advantages be included within this description and be protected by the accompanying claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The components in the drawings are not necessarily to scale relative to each other. Like reference numerals designate corresponding parts throughout the several views.

[0028] FIG. 1 illustrates an example automatic monocular vision system for one side of a vehicle inspection line.

[0029] FIG. 2 illustrates an implementation including camera arrangement of 103, processor 105 and display unit 107 with connecting cables 104 and 106.

[0030] FIG. 3 presents three example perspective views of the camera module shown in Figure 2, capturing the front, middle, and rear sections of the car body as the vehicle passes by.

[0031] FIG. 4 depicts an architecture diagram of the vision system.

[0032] FIG. 5 illustrates an example method of visual inspection and data processing, including capturing image sequences, processing the images, applying dimension prediction models to estimate and verify the dimensions of objects on an assembly line, and generating control charts to monitor dimensions for quality control.

[0033] FIG. 6 depicts a graphical user interface (GUI) for visualizing control chart.

[0034] FIG. 7 depicts a graphical user interface (GUI) for setting up templates for new measure locations.

[0035] Fig. 8 illustrates an example computing device.

[0036] Figs. 9A and 9B illustrate an example implementation of the method of Fig. 5.

[0037] FIG. 10 illustrates an example benchmark showing the improved computational efficiency of an example implementation of the present disclosure.

[0038] FIGS. 11A-11B illustrate example edge points extracted using a canny edge detection method, according to a study of an example implementation of the present disclosure.

[0039] FIG. 11B illustrates histograms of edge-points shown in FIG. 11B

[0040] FIGS. 12A-12F illustrate examples of measure-area matching, according to a study of an example implementation of the present disclosure.

[0041] FIGS. 13A-13B illustrate example edge-based image matching results according to a study of an example implementation of the present disclosure.

[0042] FIGS. 14A-14F illustrate BIW measure area matching across different perspective views according to a study of an example implementation of the present disclosure.

[0043] FIGS. 15 illustrates a template for template matching, according to an implementation of the present disclosure.

[0044] FIG. 16 illustrates example detection results using the template of FIG. 15.

[0045] FIGS. 17A illustrates a template for template matching, according to an implementation of the present disclosure.

[0046] FIG. 17B illustrates example detection results using the template of FIG. 17 A.

[0047] FIG. 18 illustrates an example of agent and environment interaction with a Markov decision process (MDP).

[0048] FIG. 19 illustrates a state space partitioned into four subspaces.

[0049] FIG. 20 illustrates detection results of a soft actor critic agent, according to a study of an example implementation of the present disclosure.DETAILED DESCRIPTION

[0050] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art. Methods and materials similar or equivalent to those described herein can be used in the practice or testing of the present disclosure. As used in the specification, and in the appended claims, the singular forms “a,” “an,” “the” include plural referents unless the context clearly dictates otherwise. The term “comprising” and variations thereof as used herein is used synonymously with the term “including” and variations thereof and are open, non-limiting terms. The terms “optional” or “optionally” used herein mean that the subsequently described feature, event or circumstance may or may not occur, and that the description includes instances where said feature, event or circumstance occurs and instances where it does not. Ranges may be expressed herein as from "about" one particular value, and / or to "about" another particular value. When such a range is expressed, an aspect includes from the one particular value and / or to the other particular value. Similarly, when values are expressed as approximations, by use of the antecedent "about," it will be understood that the particular value forms another aspect. It will be further understood that the endpoints of each of the ranges are significant both in relation to the other endpoint, and independently of the other endpoint. While implementations will be described for automobile manufacturing qualitycontrol, it will become evident to those skilled in the art that the implementations are not limited thereto, but are applicable for other types of computer vision.

[0051] Implementations of the present disclosure include improvements to gap and / or flush measurements that can be used for manufacturing quality control. Computer vision systems used in manufacturing quality control are limited by lighting, the movement of the manufactured objects being measured, and / or the attributes of the camera systems used (e.g., number, type, position(s), etc.). Gap and flush measurements are still commonly performed manually with specialized instruments (e.g., feeler gauges or dial gauges), which can require considerable labor and time. Implementations of the present disclosure can automatically and noninvasively measuring dimensions on body-in-white (e.g., unfinished and / or unpainted car bodies), including gaps and flushes, thereby reducing the labor costs associated with manual measurement. Conventional machine vision systems may also be too slow to automatically measure gaps in real time. Implementations of the present disclosure allow for parallel processing steps (e.g., template matching) which can dramatically improve processing speed. FIG. 10 illustrates an example benchmark showing the improvement of an example implementation with template matching “TM” when running on a GPU as compared to a CPU or a mix of CPU and GPU.

[0052] An example vision measurement system includes a single camera positioned to face the inspection line. As a vehicle moves through the line, the camera can capture approximately 30 images, ensuring comprehensive coverage of the vehicle's details.

[0053] The disclosed vision measurement system can measure the dimensions of gaps and flushes for body-in-white in the assembly line. This system can accurately detect the measurement area and generate dimensional values in real-time with GPU acceleration.Implementations of the present disclosure can further include Bayesian optimization methods to optimize the parameters of image processing, thereby enhancing the accuracy of edge detection and measure accuracy. The example detection and tracking method is based on exhaustive search with a hierarchical level. However, the present disclosure contemplates that a Markov Chain Process can be used to reduce the searching cost by leveraging the historical trajectory. Alternatively or additionally, to predict the gap and flush, the application of sequence models like RNN (or LSTM) or Transformers can be used.

[0054] Dimension inspection can be used in the manufacturing process to identify the non-conformities at an early stage. Gap and flush measurements and inspections are used to ensure the proper fit and finish of the body-in-white assembly. According to Lee et al. (2008), defects arising from significant gaps and flush variations between the car body and variouspanels (such as doors, hood, and others) can lead to undesirable issues like noise, inadequate insulation, or water leakage. These issues can progressively diminish the car's quality and performance. Therefore, gap and flush measurements serve as criteria in quality control to identify the defects during assembly process.

[0055] The gap refers to the horizontal separation between two surfaces, while flush represents the vertical alignment between two surfaces in the orthogonal direction. The traditional methods to measure gap and flush are still operated manually in an assembly line with specific instrument, such as feeler gauges or dial gauges, which is not easy to obtain continuous data, and it requires considerable labor cost and inspection time. To address this issue, several visual inspection systems have emerged. Portable laser triangulation-based devices have been on the rise in the market and research, such as GapGun Pro [4], LMI LaserGauge [5], and In-sight Laser Profiler [6], smartphone integrated laser triangulation system (Minnetti et al.,2020; Pham et al.,2021). Unfortunately, the lasers used in vision inspection systems can be harmful to the human body, particularly the eyes. Additionally, laser sensors are highly sensitive to the surface finish and reflectivity of the car body (Kosmopoulos et al, 2001). Surface irregularities and the angle of the laser beam can pose challenges to accurate measurements. Besides, these devices still work in contact mode and need humans to hand-hold. Kholkhujaev et al. (2022) and Tran, T., & Ha, C. (2018) build noncontact measurement systems that are integrated into a robotic arm to eliminates the potential for operator, but the robotic arms need for a clear and defined workspace to operate in which is not always possible in production line.

[0056] Kosmopoulos et al. (2001) constructed a stereo vision measurement system that uses two infrared LED lamps to highlight the edges of the gap and flush through specular reflection, and then computes the distances between the highlighted edges to determine the gap and flush measurements. However, the stereo vision system complicates the implementation process due to the requirement of two synchronized cameras, necessitating significant calibration and adjustments to the production setup.

[0057] The present disclosure can resolve the aforementioned problems by providing a flexible and easy-to-setup non-contact monocular vision system and associated methods for the automated measurement and inspection of gaps and flushness on car bodies. This present disclosure recognizes that advancements in computer vision, imaging technology, and image processing now enable the creation of a highly efficient and accurate system for measuring gaps and flushness in automotive assembly lines.

[0058] An example implementation includes (1) a computer vision system equipped with a high-resolution monocular camera to capture sequences of images as vehicles pass by on the assembly line, (2) a robust and fast image processing unit that utilizes cutting-edge algorithms to filter out the qualified images, detect measurement areas, compute each dimension of gaps and flushness, and compare these measurements across various components of the vehicle, (3) control charts to monitor the dimensions of gaps and flushness for real-time quality control, (4) a GUI-based vision module to visualize various dimensions and control chart, and allows for parameter tuning, providing an intuitive interface for operators.

[0059] This system is designed to be highly adaptable and can be seamlessly integrated into existing assembly lines without interrupting production, thus ensuring ongoing quality control while keeping the assembly process uninterrupted. Its non-contact nature reduces labor costs and enhances efficiency, making it a cost-effective solution for automotive manufacturing. The image processing unit, accelerated by graphics processing units (GPUs), can process image sequences at high speeds to keep up with the fast pace of modern assembly lines while maintaining precise measurement accuracy. The system can be scaled to handle different types of vehicles and assembly line configurations, making it versatile for various manufacturing needs.

[0060] The present disclosure includes a non-contact monocular vision system and associated methods designed for the automated measurement and inspection of gaps and flushes between the car body and its fitted panels. It can include a comprehensive vision system, which integrates hardware and software, and advanced image processing methods extract pixel measurements from images and leverage statistical models to accurately predict gap and flush dimensions, and control chart to monitor the trend and deviations in these dimensions during inspection. Gap and flush measuring can be used to control the assembly quality.

[0061] An example implementation can include the following components:1) A flexible and easy -to-setup image capturing unit with high resolution camera and integrated light.2) A plug-in and fast processing modules for:a. Image filteringb. Measure area localizationc. Edge detection and measure point detectiond. Pixel distance computinge. View group classificationf. Gap and flush dimension prediction3) GUI-based vision module for:a. Dimension data presentingb. Control charts displayc. Template configurationd. Model Tuning

[0062] An overview of the automatic monocular vision system for one side of a vehicle inspection line is shown in FIG. 1. A high-resolution camera (103) is positioned at a distance d from the inspection line (101), capturing a sequence of images as the vehicle (102) passes by. In practice, approximately 30 images per vehicle are recommended to ensure accurate predictions. The frame rate is determined by dividing the total time the vehicle is visible in the camera's view by the desired number of captured images. All images will be processed by the processors (105). The processor(s) 105 can be implemented using any / all of the components described with reference to the computing device 800 of FIG. 8. The processed results, including measure coordinates, pixel distance and gap and flush dimensions, will be combined with vehicle identification data and measure location information. This combined data with control charts will then be presented on the GUI (107). Connections, such as cables or other network connectors (104 and 106), are used to connect the camera to the processor and the processor to the GUI.

[0063] In the example implementation shown in FIG. 2, a single high-resolution camera (201) with a protective case (202) is mounted on a podium (206) with a support arm (205). The processor with GPU unit (207) is set inside of the podium (206). The display unit (204) is mounted on top of the podium (206). Cable (210) connects the display unit and processor unit. Cable (211) connects the camera and processor unit. The camera is positioned at a height of 1.20 m from the ground, ensuring that the camera angle is perpendicular to the surface being inspected. FIG. 3 provides an example of the camera's view when the vehicle appears.

[0064] FIG. 4 depicts the architecture and design of a vision inspection system, which is well-suited for a non-disruptive assembly line focused on gap and flush quality control. In this example implementation, a capture unit (400) records a sequence of images (403) which are then transmitted to a processor unit (410). This unit processes the images to estimate the dimensions of gaps and flushes. The estimated dimension data is subsequently sent to the quality control unit (420) for both overall and individual dimension assessments. Both the captured images and processed data are stored in storage unit (430). The real-time computeddimension data and historical data, along with control charts, are displayed on the display unit (440). This unit also facilitates the live streaming of the inspection line, enables data search, and allows configuration of image processing and model tuning.

[0065] The capture unit includes a high-resolution camera (401) that captures a sequence of images (403) as each vehicle passes by. This camera is coupled with a light source (402) to illuminate the target area during image capture.

[0066] The images transmitted to the processor unit (410) can undergo three steps: step 411 (Image Filtering), which extracts qualified images; step 412 (Image Processing), which further processes these qualified images to compute the pixel distance of measure locations; and step 414 (Dimension Prediction), which estimates the physical units of gaps and flushes from pixel distances for each location. GPUs (413) is employed to accelerate computing for step 411 and step 412.

[0067] The dimensions estimated in step 414 can be transmitted to the quality control unit (420), which conducts both overall control assessments (421) and individual dimension assessments (422). These assessments utilize Hotelling’s T2 charts, calculated using measurements from all locations along each inspection line. Furthermore, univariate Exponentially Weighted Moving Average (EWMA) control chart for each dimension (gap and flush) at every location are used to monitor quality. Upper and lower bounds are computed for each control chart to detect outliers (423). Corresponding warning messages are then generated and displayed on the Graphical User Interface (GUI).

[0068] The storage unit (430) comprises two types of storage solutions: file storage (432), which stores the captured, qualified images (431), and an SQL database (437), which archives the process log (433), image-derived pixel distances (434), estimated dimension data (435), and quality control check data (436). This database also integrates vehicle identification and measurement location data.

[0069] The user display unit (440) presents data from the real-time processing unit (410) and the quality control unit (420), as well as historical data from the storage unit (430). This includes image-derived pixel distances (434), estimated dimension data (435), and quality control check data (436). Additionally, this unit streams live footage from the inspection line, provides a data search function for root cause analysis of potential quality control issues, and enables configuration of image processing and model tuning (444). These features allow the system to adapt and extend to user-defined new measurement locations and refine the model as the training data expands.

[0070] Once the hardware setup is finalized and the templates for the specified measurement locations are in place, the vision system can be activated to capture images and perform measurements at the designated spots, following the process depicted in the flowchart of FIG. 5. The image sequence captured in step 500 can be processed by: image filtering at step 510, image processing for qualified images at step 520, and dimension prediction for gaps and flushes at each measurement location at step 530. To enhance efficiency, parallel computing can be employed for image filtering (step 510) and / or image processing (step 520) by specific locations.

[0071] During step 510, the captured images can undergo two processes. Initially, a ResNet image classification model can be trained and applied to predict the label of each image based on the vehicle's appearance in the camera view in step 511. The labels assigned are 'front', 'middle', and 'back' of the vehicle, or 'discard' if they are occluded by humans or other objects, exhibit motion blur, or fail to adequately cover the vehicle. Images are compressed to low resolution to speed up processing while maintaining sufficient classification accuracy. Many applications might use other artificial neural network frameworks for image classification. Images labeled as 'front', 'middle', and 'back' are considered valid and are then processed in step 512 to check the vehicle's movement status from the last frame to the current frame within an acceptable time window, At. If the vehicle remains stationary for a duration At, indicating a halted inspection line, we will reduce the computational load for the static inspection line.

[0072] To detect vehicle movement on the inspection line, we calculate the frame difference, AZ, as the median blur of the difference between consecutive frames, denoted as (1) and then apply additional to the significant difference pixels identified by threshold filtering to get AZmaskwith, (2),

[0073] where the threshold T, defined using a weighted sum of neighborhood values with a Gaussian window (block size = 7, 9, 11 or other value). Then Imaskis refined using the THRESH BINARY INV and a morphological close operation to clean up the mask, enhancing the distinction between moved and stationary areas. Finally, a robust indica movement ratio (3), is calculated by dividing the count of non-zero pixels in ljnask by the total number of pixels in the mask, providing a quantitative measure of movement,

[0074] If the move ratio exceeds a predefined threshold, the vehicle is considered to be moving, and the current frame is then transferred to image processing at step 520 for further analysis. If the ratio does not meet the threshold, the frame is ignored, and optionally no further processing occurs at step 513. This method offers a robust measure of movement by focusing on changes between images, effectively identifying and quantifying vehicle motion on the inspection line.

[0075] In the image processing step 520, the system first localizes the measurement area — or the Region of Interest (ROI) — in each image in step 521 by utilizing template matching algorithms and / or reinforcement learning methods. Subsequently, a Canny edge detection method, refined to consider pixel brightness, is employed to precisely extract the double edges of gaps on the car body in step 522. This facilitates a more accurate localization of the measurement points in pixel coordinates at each location at step 523. Using these pixel coordinates, the pixel distance between the double edges, which indicates the dimensions of gaps and flushes, is computed at step 524. Prior to proceeding to dimension prediction at step 530, outlier detection at step 525 and subsequent outlier handling at step 526 are carried out to ensure the reliability and accuracy of pixel distance values. GPUs is employed to accelerate step 511 and step 521.

[0076] Example methods of Template Matching at step 521:

[0077] At step 521, numerous deep learning models are available for the task of ROI (Region of Interest) detection. However, these models are typically supervised, necessitating substantial manual effort to label the ground truth of the measurement area for training the neural networks. To circumvent this challenge, we employ an unsupervised template matching algorithm to localize the target measurement area, which is then accelerated with GPUs for image-based operations to achieve the real-time monitoring in production.

[0078] Matching approaches are conducted with GPU accelerations: area-based and histogram of oriented gradients (HOG)-based. For area-based approach, sum of absolute difference (SAD) or normalized cross correlation (NCC) are commonly used to measure the degree of similarity or dissimilarity between template and target patch in the source image. Normalized cross correlation (NCC) is more robust to the light variance. For HOG-based approach, SAD of HOG descriptor are computed to measure the similarity between template and target patch in the source image. Example 1, hereto provides additional examples oftemplate matching and reinforcement learning algorithms that can be used in implementations of the present disclosure.The formulas to calculate SAD and NCC between template (T) and target patch (P) in the source image (I) are shown below (4) (5). NCC is widely used as its invariance to linear brightness and contrast variation (Hashemi, et al. 2016).

[0079] Where P(fc, Z) represents the pixel intensity at pixel coordinate (k,l) in target path and T(i, ) represent the pixel intensity at pixel coordinate (i,j) in the template. aP, aTdenote the standard deviation of pixel intensity values in patch.

[0080] Histogram of Oriented Gradients are a feature descriptor designed to effectively capture the structure and appearance of a target area by analyzing the distribution of intensity gradients or edge directions in localized portions of an image. HOG method involves dividing the image window into small spatial cells, computing weighted histograms of gradient magnitudes for each cell, and then applying contrast normalization to these histograms over the blocks.

[0081] Each cell accumulates a 1-D histogram of gradient directions or edge orientations based on the pixels within the cell, with the gradient magnitudes serving as weights. The orientation bins within the histogram are evenly spaced over a 0-180 degree range for unsigned gradients, or a 0-360 degree range for signed gradients. This approach achieves a balance between feature invariance (to translation, scale, and illumination) and robustness to nonlinearity. After computing the HOG descriptors, a SAD is calculated between the HOG descriptor of a template (T) and each candidate patch (P) in the source image (I) to find the best matching patch.

[0082] Template Matching with Edge-Based Optimal Transport.

[0083] The present disclosure includes a method for template matching based on edge-points and optimal transport and an improved version of Procrustes Aligned Optimal Transport (PAOT). The proposed method accounts for geometric transformations, including rotation, scaling, and projection, during the matching process, and can be applied in conjunction with the implementations of the present disclosure described herein. The method jointly learns both the optimal coupling between edge point sets and the geometric transformations (i.e. rotation, scaling, and projection) needed to align the sets. By integratingthese elements, the proposed method bridges the gap between correspondence learning and geometric alignment, offering a more robust solution for image matching.

[0084] Implementations of the present disclosure include improvements to Optimal Transport based methods of optimization for image matching.

[0085] Optimal transport is particularly effective for ensuring reliable comparisons or detections in quality inspection task as it emphasizes the global structural distribution of pixels or features, rather than relying on individual pixel or feature comparisons alone.However, much existing Optimal Transport (OT) alignment method research focuses on pixel-level or distribution-level matching and does not inherently account for structural information, such as edges, textures, or geometric features within images. Incorporating edge structures is essential for achieving accurate and precise image matching, as they play a critical role in capturing the spatial and contextual relationships in visual data.

[0086] Edge-Based OT Image Matching

[0087] The Edge-Based OT image matching begins with extracting edge points with edge detection methods detailed in step 522, resulting in source and target point sets, X and Y, from the test and reference images, respectively. FIGS. 11 A and 1 IB illustrate an example of edge points extracted using the Canny edge detection method along with the corresponding histograms of edge-points for each coordinate. These extracted edge points are then matched efficiently using optimal transport techniques.

[0088] The source and target distributions, gsand / / t, are derived from the edge points of image Zxand Z2. Given the spatial distribution of gsand, along with a pairwise cost function C X X y -> Uk+defines the cost (or dissimilarity) of matching each pair of points between the two images, the goal is to minimize the transportation cost required to align gswith gt. The Regularized OT problem for edge-based image matching is formulated as follows:mm <r, C) - eH(r), (3.10)

[0089] where the goal is to find the optimal transport plan T that minimizes the total cost (F, C), while incorporating entropy regularization eH(F) to ensure smoothness and computational stability. Here H(F) ■= —Tij log Z]y.

[0090] The problem can be efficiently solved using the Sinkhorn-Knopp algorithm, which iteratively refines the transport plan to determine the best optimal coupling between the edge points of Image Zxand Image Z2. This coupling aligns the edge distributions of thetwo images while minimizing the dissimilarity encoded in the cost matrix C. The solution takes the form,u = — and v = (3.9) Kv KTUV 7_c, where K is a kernel in the form of e e. The Sinkhorn-Knopp algorithm involves iterations that reduce to straightforward matrix-vector multiplications, making it highly compatible with GPU acceleration and enhancing computational efficiency.

[0091] Optimal Transport (OT) has notable limitations for image matching, as it primarily focuses on establishing correspondence without explicitly addressing geometric transformations such as rotation, scaling, or projection.

[0092] The present disclosure introduces Procrustes- Aligned Optimal Transport (PAOT), a method that combines Procrustes analysis with Optimal Transport. This approach jointly “learns about” or derives both the geometric transformations and the optimal correspondences between two images, enabling more effective and accurate image matching.

[0093] Edge-Based PAOT image matching

[0094] Given two sets of edge points X G Ukdxnand Y 6 Ukdxm, derived from two images / xand / 2, PAOT involves two levels of decision-making: 1) find transformations PyG Ukdxdto project Y into the space of X, and learn a scaling factor k that minimize the distance d(X, kPyY Here, d(v) denotes the distance function, typically chosen as the squared Euclidean distance. 2) Once Y = kPyY is obtained, learn the pairwise correspondence matrix T G IRnxmto align X and Y.

[0095] To simultaneously learn the transformation and correspondence between two sets, the optimization problem can be formulated as follows:s. t. r G n, Py Py = id, where y is a balance parameter and the constraint PyPy= Idensures that Pyis orthogonal(i.e., it represents a rotation and / or reflection). The objective function includes two cost components.gqand g2, which address the following aspects: 1) jointly learning the correspondence, transformation and scaling factors; and 2) preserving the geometric structurewhile learning the transformation matrix. The joint optimization problem can be simplified as:

[0096] The problem is component- wise convex in P, k and F, respectively, and can be solved in closed form with Three-Block Gauss-Seidel Method (Jin et al., 2021).

[0097] Template Matching with Edge-Based PAOT.

[0098] PAOT can be integrated into the template matching framework to enhance measure area detection and handling variations in rotation, scale, and other transformations as demonstrated in Algorithm 1.

[0099] Algorithm 1 Template Matching with PAOTInput:Template image and search image T E Ukwxftand I E 1WXHInitialize:S* <- 0;Best match location (x,y)* <- (0, 0)EdgeT <- Canny detector (F)I = 0Loop x E [1, W — w + 1]:Loop y E [1, H — h + 1]:R «- I'[x: x + w,y: y + h]EdgeR <- Canny detector (R)Compute earth mover distance d <- PAOT(EdgeT, EdgeR)If d < S* ThenUpdate S* «- dUpdate Best (x,y)* <- (x,y)End IfEnd LoopEnd LoopReturn Best match location (x, *and si:

[0100] Example methods of Reinforcement Learning for localization and tracking at step 521:

[0101] The present disclosure introduces template matching as an alternative approach for target detection in data-limited contexts. However, most methods perform a global search across every video frame, which is computationally expensive and impractical for real-time vision inspection applications. The present disclosure further includes reinforcement learning (RL) techniques and Markov Decision Processes (MDP) for sequential decision-making, can be used to detect and track target areas or objects in vision inspection in limited-data and even unsupervised scenarios. The method can include learning an optimal tracking policy by modeling it as an MDP, aiming to maximize the reward of tracking accuracy. RL allows systems to handle occlusions, abrupt movements, and changing environments by learning adaptive policies that anticipate and recover from these challenges.

[0102] FIG. 18 illustrates Agent and environment interaction with MDP. The agent and environment engage in a continuous cycle of interaction: the agent selects actions, and the environment responds by presenting new states or situations. The goal of the agent is to optimize its cumulative performance over time by strategically choosing its actions.

[0103] An MDP is defined as a tuple {S, A, P, R, y), where S is a finite set of states, and A denotes a finite set of actions available to the agent. The state transition probability matrix P encapsulates the dynamics of the environment by specifying the probability of transition to state s' from state s when action a is chosen:

[0104] Long-term benefits are captured by the concept of return, denoted Gt, which represents the cumulative discounted reward received from time t onward,, where R is the return function which assigns expected immediate rewards to stateaction pairs.

[0105] The objective of solving MDP is to identify an optimal policy that maximizes the expected return. Parameterized by 0, the policy TI S ■-> A defines the probability of selecting action a given the current state stp a\st) = 7igst, ). (4.4)

[0106] The method can formulate the problem of target tracking in a video sequence:v=Ui> ^2< ■■■ v) with a Markov Decision Process MDP {S, A, P, R, y). State space S can be partitioned into four subspaces as illustrated in FIG. 19:1) Initially Detected Sinit, where the target is first detected or reinitialized after lost, 2) Confidently Tracked Strack, where the target is successfully and confidently tracked,3) Uncertainly Tracked, Strackjow, where the target is tracked with uncertainty or low confidence,4) Lost, Siost, where the target is no longer visible or trackable in the current frame.

[0107] State st. Each state stcontains information that represents the status of the tracking process at time step t. A typical state for the tracking problem includes follow elements. Image observation features f(It}, previous target location in last frame,the appearance model (template)of target in lt-1, for instance, with edge points; motion information, vt= (vx, vy). Additional metrics, such as optimal transport (OT) distance and confidence scores, provide contextual information for evaluating tracking accuracy, e.g., OT transformation and EMD between edge_point on time t-1 and transformed edge points t-2.

[0108] Action atG A. At each t, the agent selects an action atfrom a set of options that have both discrete and continuous components, at= (adiscrete t,ac<mtinous,t}'-

[0109] The discrete action aaiscrete.t is selected from three possible choices: tracking with moving bounding box and updating the template, tracking with moving bounding box and ignoring the update, declaring the target as lost, and initializing or reinitializing the bounding box through a global search. Formally,adiscrete, te(Track & Update, Track & Ignore, Declare Lost, Initialize}. (4.7)

[0110] The continuous action, acontinous t, which specify the precise adjustments to refine the position of the bounding box when adiscrete tis Track.[OHl] When the discrete action adiscrete tis Declare Lost or Initialize, the continuous component is set to acontinous t= (0, 0), indicating no positional adjustments are necessary during these actions.

[0112] Reward rtdenotes the immediate reward received after executing action atto state St. The reward at time step t, rt, is the sum of two main components: an action-based reward and an appearance-based (similarity) reward.

[0113] The action-based reward is determined by the chosen discrete action at time t and whether the target is correctly tracked or lost (set as 0 or 1). The appearance-based reward, which evaluates the similarity between the target's appearance in consecutive frames, is determined as(4-14)

[0114] where stis the similarity metric, (e.g., Earth Mover’s Distance (EMD) or Normalized Cross Correlation NCC) between the appearance features of bounding box ltand Here the scale factor of w and w2determine how the reward is weighted when the similarity meets a specified threshold, as well as when it does not. The parameter b shifts the similarity reward to the appropriate scale. Use EMD as an example:st= -EMD Edgt, Edgt-1). (4.15)

[0115] To address the Markov Decision Process (MDP) problem in target area tracking within dynamic environments, Soft Actor-Critic (SAC) algorithm form Haarnoja et al. (2018) is utilized to learn an optimal policy that governs the decision-making process for tracking the target, specifically determining the appropriate movements of the tracking bounding box at each step.

[0116] Example methods of Edge Detection (522):

[0117] In step 522, the Canny method is initially employed to detect the edge structure of the target measurement area. Subsequently, these edges are enhanced by considering the brightness, and morphological operations are applied to the edge images to remove noise and isolated edge points. The canny method proposed by Canny (1986) is derived based on three criteria of signal-to-noise ratio (SNR) maximization for good edge detection, good localization, and single-edge response. The general process of Canny edge detection can include the following four steps:

[0118] Step 1: Smooth the image with the two-dimensional Gaussian functions. Let I(x,y) denote the image and G(x,y) denote the 2D Gaussian function as (6):

[0119] , where a is the standard distribution and the parameter to control the smooth level. The method can include forming the smoothed image Zs(x,y) by executing the convolution operation (denoted with symbol *) to the image defined as below (7):(x> y) = G (x, y) * I (x, y) (7)

[0120] Step 2: Compute the gradients including the magnitude and direction of each pixel. The magnitude and direction can be computed as (8) and (9):, and0(x,y) = tan-1(y^) (9)

[0121] , where gx(x, y) and gy(x,y) respectively describe the first derivative in the horizontal direction and vertical direction. Note that the magnitude Ms(x, y) and direction 0(x,y) are arrays that have the same size as the smoothed images from Step 1. Step 1 and Step 2 can be combined into one step by applying a first derivative of a Gaussian kernel.

[0122] Step 3: Suppress non-maximal gradients, so-called nonmaximal suppression. Typically, the gradient magnitude image Ms(x, y) contains wide ridges around local maxima. This step is to obtain thin and accurate location edges. For any arbitrary pixel (x,y) in 0

[0123] , the method can define 8 gradients directions in a 3^3 region centered at this pixel. The direction of this pixel is 6k. If the magnitude of this pixel, Ms(x, y), is larger than its two neighboring pixels along the direction 6k, then it will be chosen as a candidate edge point, otherwise, it will be suppressed. Therefore, a nonmaximal suppressed image with candidate edge points is yield.

[0124] Step 4: Detect and link edges with double thresholding and connectivity method. Single thresholding on the nonmaximal suppressed image from step 3 can cause broken edges. To improve this situation, the canny method uses double thresholding, so-called “hysteresis thresholding”: a lower threshold Tl and a higher threshold Th. If a pixel whose gradient magnitude is larger than Th, it will be marked as an edge point (considered as a “strong” edge point). If the magnitude is less than Tl, it will be marked as a non-edge point and will be discarded in the output image. If the magnitude is less than Th but larger than Tl, it will be marked as a candidate edge point (considered as a “weak” edge point). Theconnectivity of this point to the strong edge points will be checked to determine if it is a valid edge point.

[0125] The pixel brightness around each edge point can be carefully examined to ensure accurate measurement. It is assumed that pixels closer to the gap of the measurement area are darker compared to those on the body -in-white of the vehicle. Therefore, a brightness threshold (B thresh) is defined to select significant edge points which are in proximity to darker area (< B thresh). Additionally, morphological operations are applied to remove objects smaller than a predefined size threshold (S thresh), further refining the edge detection process.

[0126] Step 530

[0127] The pixel distances computed at each measurement location serve as input for dimension prediction at step 530 to estimate the gap and flush value at each location. The pixel distances from each image in the sequence for each vehicle are grouped into three view categories — 'left view,' 'middle view,' and 'right view' — to mimic multi-view camera perspectives based on the pixel coordinates of the measurement locations (531).Subsequently, two distinct second-order regression models for each location are employed for spatial calibration to convert the dimensions measured in pixels from each view into physical units (532). This model is formulated to represent the relationship between the dimension in pixels from different views. The central assumption is that this relationship can be modeled by the equation (10):y = Xp + E (10)

[0128] Here, X denotes the mean value of pixel distance derived from the left, middle, and right views, along with the second-order interaction terms. Including the standard deviation of pixel distances from each view group in X might further enhance the estimation accuracy, y represents the known real -world dimensions of gap and flush.?; in describes the regression coefficient of each term and E is a disturbance term describing the residual error. Then outlier detection (533) and subsequent outlier handling at step 534 are carried out to ensure the reliability and accuracy of the measurements.

[0129] Outlier handling at step 526 and step 534 includes filling in missing values and replacing extreme values that exceed the 3 -Sigma Limits by averaging adjacent frame values. According to 3-Sigma Limits rule (11), nearly all (99.73%) values lie within three standard deviations of the mean x. Therefore, if the pixel distance or estimated dimension of a measurement deviates from this range, it can be considered an outlier. Specifically, a1measurement is flagged as an outlier if the absolute difference from the mean exceeds three standard deviations (3a):|x — x| > 3 * o(x) (11)

[0130] This criterion ensures that only measurements that significantly deviate from the norm are treated as outliers, maintaining the integrity and accuracy of the data analysis. For each dimension at measure location, an individual rule is established to ensure values do not deviate significantly from the mean within that dimension. This approach is particularly effective in managing the diversity of pixel distances across different measurement locations and in accommodating varying criteria for acceptable dimensions, which may differ based on the type of dimension and location.

[0131] Once the estimated gaps and flushes are derived, control charts for the dimensions are generated for quality control purposes. The dimensions and their respective control charts are displayed on the GUI interface, as illustrated in FIG. 6. These include an overall multivariate control chart of Hotelling's T2at the top and univariate Exponentially Weighted Moving Average (EWMA) control chart for each dimension at each measurement location, displayed at the bottom. Measurement locations are identified by IDs such as 7, 8, 9, etc. The smooth factor of the EWMA control charts can be adjusted by users to analyze trends at various levels of historical data recall. Each chart features an upper bound limit (UCL) and lower bound limit (LCL) marked by a red line.

[0132] Example methods of Hotelling's T2and EWMA control chart (Step 550):

[0133] Hotelling's T2Control Chart is a method to create a single chart for the simultaneous monitoring of multivariate continuous data and determine if the data are within control limits over time. The present disclosure can use the method provided by Allen (2019 P206; MacGregor and Koutoudi, 1995) to compute Hotelling's T2charting. This system collects and computes p dimensions (gap and flush on each measure location) for n time stamps (resulting in n multivariate observations). Each observation xtcan be denoted as a (p X 1) vector of measurements on p normally distributed variables. The T2statistics for charting can be calculated using (12)T2= n(Xj — x)'-1(Xj — x)(12)

[0134] , where S’ is the unbiased pooled covariance matrix estimated from a sample of the n past multivariate observations.

[0135] The control limits can be computed as below (14) and (15),, where Fa p n-pis the upper 100 a% critical point of the F-distribution with p and n-p degree of freedom.

[0136] The exponentially weighted moving average (EWMA) control chart is an efficient control tool that can quickly detect assignable causes that signal small shifts in the process. EWMA chart is a type of control chart that uses both recent and historical information by adopting a geometrically decreasing weight scheme. The highest weight is assigned to the most recent observations, while the weight decays exponentially for the more distant observations. Assume that there is a subgroup of size n observations at each time point, xtis defined as the subgroup mean of the observations at time t. Then, the EWMA statistic ztat time t is defined as a weighted average of current observations xtand all proceeding observations zt-x:zt= Axt+ (l - A)zt-1(16)

[0137] where zt-1is the running exponential average of all proceeding observations until time point t-1, and A is the weight, called the “smoothing factor”, assigned to the most recent subgroup mean at t. The parameter A determines the memory behavior of prior data in EWMA. The small value of A such as 0.05 puts more weight on older data which makes it more sensitive to small shifts. The upper (UCL) and lower (LCL) control limits for the EWMA chart at time t are given as equation, (17):

[0138] where x and s respectively represent the estimated mean of all historical subgroups means and the common standard deviation of each subgroup. The factor L is the multiple of the subgroup standard deviation, typically set as 3 for processing control or adjustment according to Lucas and Saccucci (1990) table. If the value of ztis outside of these control limits, it is a signal of out-of-control.

[0139] At step 560, the control charts can be output. The control chart can optionally include measures of the gap between two adjacent panels of an auto body. The controller of the system can further be configured to control a panel positioning mechanism to change the gap between the two adjacent panels, so that a subsequent vehicle from the assembly line has an optimized gap between the corresponding two panels. As used herein, an “optimized” gap is a gap that meets a specification (e.g., a specific value of panel gap within a predetermined tolerance). The present disclosure contemplates that different types of panel placement mechanisms can be used for different parts of the auto body, so that optionally multiple panel positioning mechanisms can be controlled based on the output of step 560.

[0140] An additional illustration of the method of FIG. 5 is illustrated in FIGS. 9A-9B.

[0141] This vision system offers extendibility for new, undefined measurement locations. FIG. 7 displays a user interface for configuring the template for a new measurement location by drawing the bounding box around the measure area. The system assumes that the measure location close to the centroid of the bounding box will be considered the desired measurement location. The defined template will be saved to the template configuration file, which is used for image processing of new image flows.Furthermore, users can manually label the correct measure coordinates, enabling the system to tune the optimal set of image processing parameters. These parameters include edge detection thresholds, brightness thresholds, and the size of morphological operations, among others. To optimize these settings, Bayesian Optimization and Grid Search can be employed, which fine-tunes the parameters by minimizing the distance between the system-detected measure coordinates and those manually labeled by the user.

[0142] Additionally, the real-world ground truth data can be fed into the model tuning module to train the spatial calibration regression model for new measurement locations or to retrain the model for existing locations. This configuration module provides flexibility and extendibility for new measurement locations and can also be quickly adapted to other vehicle inspection lines.

[0143] It should be appreciated that the logical operations described herein with respect to the various figures may be implemented (1) as a sequence of computer implemented acts or program modules (i.e., software) running on a computing device (e.g., the computing device described in FIG. 8), (2) as interconnected machine logic circuits or circuit modules (i.e., hardware) within the computing device and / or (3) a combination of software and hardware of the computing device. Thus, the logical operations discussed hereinare not limited to any specific combination of hardware and software. The implementation is a matter of choice dependent on the performance and other requirements of the computing device. Accordingly, the logical operations described herein are referred to variously as operations, structural devices, acts, or modules. These operations, structural devices, acts and modules may be implemented in software, in firmware, in special purpose digital logic, and any combination thereof. It should also be appreciated that more or fewer operations may be performed than shown in the figures and described herein. These operations may also be performed in a different order than those described herein.

[0144] Referring to FIG. 8, an example computing device 800 upon which the methods described herein may be implemented is illustrated. It should be understood that the example computing device 800 is only one example of a suitable computing environment upon which the methods described herein may be implemented. Optionally, the computing device 800 can be a well-known computing system including, but not limited to, personal computers, servers, handheld or laptop devices, multiprocessor systems, microprocessorbased systems, network personal computers (PCs), minicomputers, mainframe computers, embedded systems, and / or distributed computing environments including a plurality of any of the above systems or devices. Distributed computing environments enable remote computing devices, which are connected to a communication network or other data transmission medium, to perform various tasks. In the distributed computing environment, the program modules, applications, and other data may be stored on local and / or remote computer storage media.

[0145] In its most basic configuration, computing device 800 typically includes at least one processing unit 806 and system memory 804. Depending on the exact configuration and type of computing device, system memory 804 may be volatile (such as random access memory (RAM)), non-volatile (such as read-only memory (ROM), flash memory, etc.), or some combination of the two. This most basic configuration is illustrated in Fig. 8 by dashed line 802. The processing unit 806 may be a standard programmable processor that performs arithmetic and logic operations necessary for operation of the computing device 800. The computing device 800 may also include a bus or other communication mechanism for communicating information among various components of the computing device 800.

[0146] Computing device 800 may have additional features / functionality. For example, computing device 800 may include additional storage such as removable storage 808 and non-removable storage 810 including, but not limited to, magnetic or optical disks or tapes. Computing device 800 may also contain network connect! on(s) 816 that allow thedevice to communicate with other devices. Computing device 800 may also have input device(s) 814 such as a keyboard, mouse, touch screen, etc. Output device(s) 812 such as a display, speakers, printer, etc. may also be included. The additional devices may be connected to the bus in order to facilitate communication of data among the components of the computing device 800. All these devices are well known in the art and need not be discussed at length here.

[0147] The processing unit 806 may be configured to execute program code encoded in tangible, computer-readable media. Tangible, computer-readable media refers to any media that is capable of providing data that causes the computing device 800 (i.e., a machine) to operate in a particular fashion. Various computer-readable media may be utilized to provide instructions to the processing unit 806 for execution. Example tangible, computer-readable media may include, but is not limited to, volatile media, non-volatile media, removable media and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. System memory 804, removable storage 808, and non-removable storage 810 are all examples of tangible, computer storage media. Example tangible, computer-readable recording media include, but are not limited to, an integrated circuit (e.g., field-programmable gate array or application-specific IC), a hard disk, an optical disk, a magneto-optical disk, a floppy disk, a magnetic tape, a holographic storage medium, a solid-state device, RAM, ROM, electrically erasable program read-only memory (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices.

[0148] In an example implementation, the processing unit 806 may execute program code stored in the system memory 804. For example, the bus may carry data to the system memory 804, from which the processing unit 806 receives and executes instructions. The data received by the system memory 804 may optionally be stored on the removable storage 808 or the non-removable storage 810 before or after execution by the processing unit 806.

[0149] It should be understood that the various techniques described herein may be implemented in connection with hardware or software or, where appropriate, with a combination thereof. Thus, the methods and apparatuses of the presently disclosed subject matter, or certain aspects or portions thereof, may take the form of program code (i.e., instructions) embodied in tangible media, such as floppy diskettes, CD-ROMs, hard drives, or any other machine-readable storage medium wherein, when the program code is loaded intoand executed by a machine, such as a computing device, the machine becomes an apparatus for practicing the presently disclosed subject matter. In the case of program code execution on programmable computers, the computing device generally includes a processor, a storage medium readable by the processor (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device. One or more programs may implement or utilize the processes described in connection with the presently disclosed subject matter, e.g., through the use of an application programming interface (API), reusable controls, or the like. Such programs may be implemented in a high level procedural or object-oriented programming language to communicate with a computer system. However, the program(s) can be implemented in assembly or machine language, if desired. In any case, the language may be a compiled or interpreted language and it may be combined with hardware implementations.

[0150] Studies were performed using example implementations of the present disclosure.

[0151] Experimental Results

[0152] Image Matching for Rotated and Scaled Images

[0153] A study evaluated the performance of standard optimal transport (OT) method and the proposed PAOT for image alignment tasks involving two regions of interest (ROIs) with differing rotation and scale. These ROIs, referred to as the source image and target image in FIGS. 12A-12F, represent measurement areas extracted from two images captured as a car body moves through the inspection line. To simulate real-world challenges, the source ROI was scaled down with a different size ratio, while the target ROI, as shown in FIGS. 12A-12F, was derived from a rotated image.

[0154] FIG. 12A-12F illustrate BIW measure area matching with rotation and scale, where FIG. 12A illustrates Source raw image, FIG. 12B illustrates edge structure of source image detected with Canny method, FIG. 12C illustrates edge points of source images in X-Y coordinates. FIG. 12D illustrates target image with applying rotation and scale, FIG. 12E illustrates edge structure of target image detected with Canny method, and FIG. 12F illustrates edge points of target images in X-Y coordinates.

[0155] In vision-based inspection, edge structures play a critical role in accurate alignment. Therefore, the alignment process is based on the edge points of the source and target images. First, the edges are extracted using the Canny method described herein, and the resulting edge images are converted into coordinate-based edge points for subsequent coupling.

[0156] FIGS. 13A-13B illustrate the edge-based image matching results using both OT and PAOT. It is evident that PAOT achieves better alignment by effectively transforming the image to account for rotation and scale differences. These results demonstrate that PAOT performs more competitively than OT in scenarios involving variations in rotation and scale between the aligned images. FIG. 13A-13B illustrate edge-Based Matching with OT and PAOT addressing rotation and scale issue where FIG. 13 A illustrates edge points correspondence with OT and FIG. 13B illustrates edge points correspondence with PAOT, where a learned geometric transformation is applied to the target edge points.

[0157] Image Matching across Different Perspective Views

[0158] In this section, the evaluation involves aligning two regions of interest (ROIs) captured from two different perspective cameras, shown in FIGS. 14A-14F. This setup tests the effectiveness of the methods in handling varying perspectives and spatial transformations. FIGS. 14A-14F illustrate BIW measure area matching across different perspective views where FIG. 14A illustrates source raw image with skewed perspective view, FIG. 14B illustrates edge structure of source image detected with Canny method, FIG. 14C illustrates edge points of source images in X-Y coordinates, FIG. 14D illustrates target image with camera facing directly at the BIW, FIG. 14E illustrates edge structure of target image detected with Canny method, and FIG. 14F illustrates edge points of target images in X-Y coordinates. The edge-based OT and PAOT approaches were implemented for image matching and incorporated into a template-matching framework, achieving successful target area detection.

[0159] Target Area Detection

[0160] The study evaluated the performance of Edge-Based PAOT algorithms for target area detection, combined with template matching. The study also compared its performance against Optimal Transport (OT), traditional pixel-level template matching algorithms such as NCC and SAD, and methods based on Histogram of Oriented Gradients (HOG) features. HOG, is a feature descriptor designed to effectively capture the gradient structure and visual appearance of a target area by analyzing the distribution of intensity gradients and edge directions in localized regions of an image. Further details about the HOG-based approach are provided herein.

[0161] In the first case, the image is rotated and captured at a slightly different scale compared to the template. In the second case, there is a perspective variance where twoimages are captured with a camera set as two different perspective angle as shown in FIGS.17A-17B.

[0162] To assess detection performance, six methods were tested and compared: two pixel-level methods (NCC and SAD), two HOG-feature-based methods (using Earth Mover’s Distance and SAD), and two edge-based methods (OT and PAOT as described in the present disclosure). The detection results are presented in FIGS. 15-16 and FIGS. 17A-17B, clearly demonstrating the advantages of using edge-based OT and PAOT methods for target detection in both cases — rotation and scale, and perspective view. FIGS. 15 illustrates a template and FIG. 16 illustrates template matching results of six methods. FIG. 17A illustrates a template and FIG. 17B illustrates a template matching result for six methods for a grid search.

[0163] The four methods, based on pixel-level and HOG features, failed to detect the target. In contrast, edge-based OT and PAOT methods successfully detected the target region with a substantially higher overlap than other methods.

[0164] Discussion

[0165] Example implementation includes the use of the Optimal Transport (OT) optimization for precise image matching, with a particular focus on edge-based matching, which improves both computational efficiency and robustness. PAOT jointly learns the correspondence between source and target edge point sets while also accounting for transformations such as rotation and scaling. The source and target images both lie in a comparable 2D domain in this problem, but it can also be extended to 3D image matching if needed.

[0166] The study also demonstrates the effectiveness of PAOT in target area detection. The experiment was presented, evaluating the proposed methods in scenarios involving rotated and scaled objects, and target area detection. The results validate the performance of PAOT, comparing it with traditional methods and highlighting its superiority in achieving precise and robust image matching. The study shows the effectiveness of OT and PAOT as powerful optimization tools for image processing, specifically for tasks requiring accurate edge-based matching and alignment in the presence of geometric transformations.

[0167] Reinforcement Learning for Adaptive Target Tracking Application and Result for Automated Inspection

[0168] Experiment Setup

[0169] The study evaluated the performance of a reinforcement learning-based tracking system, focusing on tracking a target region of the Body-in-White (BIW) during inspection. The evaluation primarily emphasizes detection accuracy and computational efficiency, for computational practicality, the image data were downsampled before analysis, and each video was decomposed into sequential frames for processing.

[0170] The Soft Actor-Critic (SAC) agent was trained using an image sequence decomposed from a video encompassing the appearance, movement, and disappearance of the target area. For evaluation, the trained SAC agent was tested on a new image sequence consisting of 50 frames that similarly capture the appearance, movement, and disappearance of the target area. Ground truth bounding boxes for each frame were manually annotated, ensuring precise evaluation of the detection performance.

[0171] Two SAC agents were, each integrating different similarity metrics — Normalized Cross-Correlation (NCC) and Optimal Transport (OT) — to assess their effectiveness in tracking performance^ Additionally, a traditional grid search was applied using template matching with pixel NCC and OT metrics for comparative purposes.

[0172] Tracking performance was evaluated using an overlap-based accuracy measure between the detected and reference regions across image frames. A detection was considered correct when the overlap exceeded a predefined threshold. Owing to the stable inspection setup and consistent imaging conditions, the bounding boxes were configured with a fixed size Based on this criterion, detection accuracy across all frames is computed as an additional performance metric.

[0173] Furthermore, computational speed, measured in frames per second (FPS), is compared, as it is a critical factor for real-time inspection applications in manufacturing environments.

[0174] Experimental Results

[0175] The study configured each agent to operate over a fixed number of time steps per episode. The SAC -based agents demonstrated superior performance compared with grid search methods in tracking image sequences, while also providing faster computation suitable for real-time applications. The SAC agent with OT achieves a detection accuracy 86.48%, significantly outperforming the grid search approach, which achieved only 32.43% detection accuracy.

[0176] In FIG. 20, the SAC agent employing OT maintained stable tracking performance through most of the sequence, with temporary fluctuations under certain irregular motion patterns. The agent demonstrated robustness by quickly recovering fromsuch cases, Training over longer sequences may further enhance adaptability to diverse motion patterns. Overall, the SAC-OT approach achieved notably higher tracking quality and computational efficiency than traditional grid search-based methods..

[0177] The example implementation includes an improved Markov Decision Process (MDP) formulation and a reinforcement learning (RL)-based algorithm for adaptive target tracking.

[0178] The results for the example implementation are compared against a baseline grid search approach. Experimental results show that SAC agents according to the example implementation outperform traditional methods by achieving higher detection accuracy and faster tracking speeds, even when trained on limited video data without supervised labels. These findings highlight the practical advantages of reinforcement learning for real-time inspection tasks with limited data.

[0179] The present disclosure contemplates that longer video training datasets can be used to improve robustness of the systems described herein.

[0180] Experimental Result for Vision Measurement

[0181] An additional study of an example implementation was performed to evaluate the accuracy of the vision system. Reference measurements were obtained using a calibrated measurement device and compared with the corresponding values estimated by the system at multiple representative locations on a set of inspected objects. Some samples were excluded from analysis due to incomplete data or image occlusion, ensuring that only valid measurements were used for evaluation.

[0182] For each geometric dimension, a regression-based estimation model was developed to capture spatial correlations among measurement regions. The model demonstrated stable and reliable performance across both gap and flush measurements.

[0183] Across all measurement locations, the average estimation error, expressed as root mean square error (RMSE), remained generally below approximately 0.5 mm for both gap and flush estimations (Table 1). This level of accuracy supports the suitability of the system for automated, real-time dimensional inspection in manufacturing environments.Measurement results using the vision system and an example regression model for 9 points.Measure Dimension RMSEpointGap 0.428126Flush 0.414029Gap 0.30557Flush 0.530022Gap 0.323271Flush 0.550152Gap 0.335193Flush 0.422949Gap 0.274541Flush 0.28853Gap 0.486559Flush 0.658986Gap 0.298791Flush 0.348868Gap 0.34081Flush 0.502967Gap 0.281065Flush 0.895433AVG Gap 0.341547AVG Flush 0.512437

[0184] The following references, in their entirety, are incorporated herein by reference.[1] Lee, S. H., Jun, C. H., Jung, J., Kim, T. S., & Lee, J. H. (2008). Identifying sources of dimensional variation affecting assembly quality of automobiles. In Proceedings of the 9th APIEMS Conference.[2] Minnetti E, Chiariotti P, Paone N, Garcia G, Vicente H, Violini L, Castellini P. A Smartphone Integrated Hand-Held Gap and Flush Measurement System for in Line Quality Control of Car Body Assembly. Sensors. 2020; 20(ll):3300. https: / / doi.org / 10.3390 / s20113300[3] Kosmopoulos, D., & Varvarigou, T. (2001). Automated inspection of gaps on the automobile production line through stereo vision and specular reflection. Computers in Industry, 46(1), 49-63.[4] Third Dimension, GapGun Pro, Third Dimension, Bristol, United King- dom, 2018.Accessed on: Dec. 23, 2019. [Online], Available: https: / / www.third.com / products / product-range / gapgun-pro[5] Linear Measurement Instruments Corporation, LaserGauge Systems: gap and flush automotive exterior, Linear Measurement Instruments Corporation, Michigan, USA, Nov.2015. Accessed on: Dec. 23, 2019. [Online], Available: https: / / www.lmicorporation.com / LaserGaugeSystems[6] Cognex Corporation, In-sight laser profiler: an easy-to-use measurement system for inspecting and verifying product dimensions, Cognex Corporation Massachusetts, USA, 2018. Accessed on: Dec. 23, 2019. [On-line], Available: https: / / www.cognex.com / products / machine-vision / 3d-laser-profilers / in-sight-laser- profiler[7] L. H. Pham, D. N. -N. Tran, J. Y. Byun, C. H. Rhie and J. W. Jeon, "A Smartphone-Based Laser Measuring System for Gap and Flush Assessment in Car Body," in IEEE Transactions on Industrial Electronics, vol. 68, no. 7, pp. 6297-6307, July 2021, doi: 10.1109 / TIE.2020.2992971.[8] J. Kholkhujaev, G. Maculotti, G. Genta, M. Gaietto and J. Inoyatkhodjaev, "Non-Contact Articulated Robot-Integrated Gap and Flushness Measurement System for Automobile Assembly," in IEEE Access, vol. 10, pp. 86528-86541, 2022, doi: 10.1109 / ACCESS.2022.3199066.[9] Tran, T., & Ha, C. (2018). Non-contact Gap and Flush Measurement Using Monocular Structured Multi-line Light Vision for Vehicle Assembly. International Journal of Control, Automation and Systems, 16, 2432 - 2445.

[0010] Hashemi, N. S., Aghdam, R. B., Ghiasi, A. S. B., & Fatemi, P. (2016). Template Matching Advances and Applications in Image Analysis (arXiv: 1610.07231). arXiv. http: / / arxiv.org / abs / 1610.07231

[0011] Dalal, N., & Triggs, B. (2005). Histograms of Oriented Gradients for Human Detection.2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), 1, 886-893. https: / / doi.org / 10.1109 / CVPR.2005.177

[0012] Zou, Z., Chen, K., Shi, Z., Guo, Y., & Ye, J. (2023). Object Detection in 20 Years: A Survey (arXiv: 1905.05055). arXiv. http: / / arxiv.org / abs / 1905.05055

[0013] Canny, J. 1986. A Computational Approach to Edge Detection. IEEE Transactions on Pattern Analysis and Machine Intelligence 8(6):679-698. doi: 10.1109 / TP AMI.1986.4767851.

[0014] Hotelling H (1947) Multivariate quality control, techniques of statistical analysis.Eisenhard H, Wallis (eds). McGraw-Hill, New York

[0015] Allen, Theodore T. Introduction to engineering statistics and lean six sigma'. Statistical quality control and design of experiments and systems. 3rdedition. Springer London, 2019.

[0016] MacGregor, J. F., & Kourti, T. (1995). Statistical process control of multivariate processes. Control engineering practice, 3(3), 403-414.

[0017] Roberts S. W. 1959. Control chart tests based on geometric moving averages.Technometrics 1(3):236— 250

[0018] Lawson, J. 2019. Phase II monitoring of variability using Cusum and EWMA charts with individual observations. Quality Engineering 31 (3):417— 29. doi: 10.1080 / 08982112.2018.1557205.

[0019] Abbasi, S. A. 2010. On the performance of the EWMA chart in the presence of two- component measurement error. Quality Engineering 22 (3). \99-2 \3. doi: 10.1080 / 08982111003785649.

[0020] Lucas, J. M. and Saccucci, M. S. 1990. Exponentially weighted moving average control schemes: Properties and enhancements. Technometrics 32(1), 1-12. doi: 10.1080 / 00401706.1990.10484583

Claims

WHAT IS CLAIMED:

1. A vision system for measuring and inspecting gaps and flushes of car body on assembly lines, comprising:a camera configured to capture a sequence of monocular images as vehicles pass along an inspection line;a controller operably coupled to the camera, the controller comprising a processor and a memory with computer-readable instructions stored thereon, that, when executed by the processor, cause the processor to:receive the sequence of images from the camera;generate a first processed image based on a first image of the sequence of images by localizing a region of interest by template matching or reinforcement learning;generate a second processed image based on a second image of the sequence of images by localizing a region of interest by template matching or reinforcement learning;estimate a manufacturing dimension based on the first processed image and the second processed image; andoutput the estimated manufacturing dimension.

2. The vision system of claim 1, wherein generating the first processed image comprises performing at least one of: image filtering, area localization, edge detection, point detection, and / or pixel distance computing on a first image of the sequence of images.

3. The vision system of claim 1 or claim 2, wherein generating the second processed image comprises performing at least one of: image filtering, area localization, edge detection, point detection, and / or pixel distance computing on a second image of the sequence of images.

4. The vision system of any one of claims 1-3, wherein estimating a manufacturing dimension based on the first processed image and the second processed image comprises detecting a movement of an object between the first processed image and the second processed image.

5. The vision system of any one of claims 1-4, wherein the manufacturing dimension comprises a measurement of a gap between a pair of panels or a measurement of a flush joint between a pair of panels.

6. The system of any one of claims 1-5, wherein the camera comprises an integrated light.

7. The system of any one of claims 1-6, wherein the camera is positioned at a predetermined distance from the inspection line and at a predetermined height, with images captured at defined intervals to ensure comprehensive coverage and accurate measurement of gaps and flushes.

8. The system of claim 1, wherein integrated lighting is configured to illuminate a target area's edge structure9. The system of claim 1, wherein the memory contains further instructions configured to:execute image filtering to select qualified images based on an appearance, clarity, or movement of a vehicle in the sequence of images, from a last frame of the sequence of images to a current frame of the sequence of images within a predefined window;perform advanced image processing techniques to localize a measure area, extract an edge structure of measure area, determine a measure coordinates of each measure point, and compute a pixel distance between two measure points; and implement an outlier handling method to fix missing values and extreme outliers in pixel distances from each image.

10. The system of any one of claims claim 1-9, further comprising a GPU, wherein the processor is configured to generate the first processed image and the second processed image by the GPU, and wherein the processing of the first processed image and the second processed image are performed in real time.

11. The system of any one of claims 1-10, wherein the estimated manufacturing dimension is based in part on a regression model configured to convert pixel distances into physical units of gap and flush dimensions.

12. The system of any one of claims claim 11, wherein the system further includes a panel positioning mechanism and the manufacturing dimension comprises a gap between a first panel and a second panel, and wherein the memory contains further instructions configured to: control the panel positioning system to optimize the gap between the first panel and the second panel.

13. The system of any one of claims claim 1-12, further comprising a GUI configured to support live streaming of the inspection process, provides data search capabilities for root cause analysis, facilitates real-time monitoring of dimensions and control charts, and enables configuration of processing parameters and model tuning.

14. A method comprising:capturing images of a manufactured object using a camera as the manufactured object passes along an inspection line;processing the images to derive a plurality of dimensions of the manufactured object;converting the dimensions in pixels to physical measurements by a prediction model for spatial calibration comprising a second-order polynomial regression; andcontrolling an assembly line based on the physical measurements or tuning an image processing parameter used to process the image based on the physical measurements.

15. The method of claim 14, wherein the step of processing the images comprises using a Neural Network framework such as ResNet, CNN, or Yolo to classify images based on vehicle sections and a movement detection method designed to detect movement with a robust move ratio indicator, ensuring that only pertinent images are processed.

16. The method of claim 14 or claim 15, wherein step of processing the images further comprises:using GPU-accelerated template matching to localize a measure area; extracting an edge structure of the measure area by an enhanced Canny edge detection using brightness and improved morphological operations; and detecting a precise measure point coordinates based on proximity to a centroid of the detected measure area.

17. The method of any one of claims 14-16, wherein dimension prediction is performed using a second-order regression model to convert pixel measurements into physical units. This method incorporates a technique that mimics multi-view camera perspectives by categorizing sequential images into different view groups based on measure coordinates, thereby enhancing an accuracy of dimension predictions, wherein a mean and standard deviation of a pixel distance in each group, along with a second- order interaction term, are fed into the regression model as input features.

18. The method of any one of claims 14-17, further comprising using outlier detection techniques to verify a reliability of pixel distance values and measurements.

19. The method of any one of claims 14-18, wherein controlling the assembly line comprises using control charts with a combination of overall Hotelling T2and univariate EWMA control charts for ongoing quality assurance.

20. A non-transitory computer-readable medium storing having instructions stored therein, wherein execution of the instructions by a processor, causes the processor to: receive a sequence of images from an imaging device;generate a first processed image based on a first image of the sequence of images by localizing a region of interest by template matching and / or reinforcement learning; generate a second processed image based on a second image of the sequence of images by localizing a region of interest by template matching and / or reinforcement learning; estimate a manufacturing dimension based on the first processed image and the second processed image; andoutput the estimated manufacturing dimension.

Images