Visual invoice mapping for object repair

Abstract

According to an aspect, there is provided systems and methods for automatically reviewing an invoice for repairing an object. The method includes receiving a representation of the invoice, processing the representation of the invoice to extract costs, mapping the costs onto at least one of one or more impacted components and one or more repair operations, comparing the costs to expected costs based on the at least one of one or more impacted components and the one or more repair operations, and transmitting or storing a report on the comparison. According to an aspect, there is provided systems and methods for providing a reliability pipeline to machine learning predictions. The method including processing data in redundant steps to predict a result, determining a likelihood estimate that the prediction is wrong, and flagging the prediction for review if the likelihood is above a threshold.

Description

VISUAL INVOICE MAPPING FOR OBJECT REPAIRFIELD

[0001] The present disclosure generally relates to the field of computing platforms, artificial intelligence, computer vision, and image processing. In particular, this disclosure relates to systems and methods of processing images of invoices to estimate repair costs.INTRODUCTION

[0002] Fleet Management Companies (FMCs) face several operational challenges that can impact their ability to manage fleet condition and performance while minimizing costs. At the core of these challenges may be the limited ability to monitor and assess vehicle conditions effectively. Failure to do so can ultimately result in reactive expenditures to address unexpected repairs, inflated costs, and suboptimal decisions for fleet renewal. Moreover, the absence of clear oversight can expose FMCs to potential fraud, particularly in the allocation and misuse of maintenance budgets, increasing the risk of financial discrepancies.

[0003] While it may be optimal to have an independent expert assess the vehicle to ensure that repair costs are reasonable, not all drivers may be experts in assessing vehicles and waiting for an expert in whatever region the vehicle is in may cause costly delays. Skipping such an assessment might mean spending unreasonably high amounts on repairs or shopping around which again may cause costly delays.

[0004] Invoices themselves can be confusing to read and report. Accordingly, it may be difficult for the drivers to accurately input all the relevant information in any reimbursement claim or easy for them to falsify. It may be challenging to automatically process invoices due to complexities such as multi-page tables, diverse table formats, and inconsistent formatting between repair shops.

[0005] Improvement in the field of invoice analysis is desirable.SUMMARY

[0006] Described herein are structured extraction methodologies called Visual Invoice Mapping (VIM) to analyze invoices for object repair (e.g., automotive body shop invoices for vehicle repair). VIM can leverage automated tools, such as Visual Language Extractors, VisualLanguage Models, and other Machine Learning methods to iteratively validate and enhance data extraction. The systems and methods described herein can further reconcile the invoice analysis with an assessment of the object (e.g., visually detecting defects in the object using, for example, computer vision).

[0007] Provided herein are systems and methods to automatically and consistently process invoices. Such solutions can aid in the prediction of costs of repairs, such as when damage may occur, and how much it may cost to repair (e.g., based on the region in which repairs are sought). The systems and methods described herein may further be used in conjunction with repair assessment reports to compare the observed damage to the object (e.g., the vehicle) to the repairs presented / proposed on the invoice to quickly approve allowable invoices and avoid costly delays. The systems and methods described herein may also be used to aid in the processing and reporting of invoice claims to a central authority. Such a process may assist, for example, drivers in getting reimbursements in a timely fashion and may reduce rates of fraudulent claims.

[0008] Applications of the present systems and methods may include inspection of any object type made up of parts, including, but not limited to merchandise, industrial equipment, aircraft parts and components, construction equipment and machinery, medical equipment and devices, electronic devices and components, furniture and fixtures, agricultural machinery and equipment, marine vessels and components, manufacturing machinery and equipment, power generation and distribution equipment, scientific instruments, equipment, etc. Other applications may include, for example, infrastructure (e.g., roadways, bridges, energy infrastructure, dams, buildings, etc.) particularly when located in remote areas but controlled by a central authority. While the following will describe vehicles, it may apply to any object that may require repairs or maintenance.

[0009] According to an aspect, there is provided a system for automatically reviewing an invoice for repairing an object. The system includes a server having non-transitory computer readable storage with executable instructions for causing one or more processors to receive a representation of the invoice, process the representation of the invoice to extract costs, map the costs onto at least one of one or more impacted components and one or more repair operations using a mapping engine, compare the costs to expected costs based on the at least one of the one or more impacted components and the one or more repair operations, and transmit or store a report on the comparison.

[0010] In some embodiments, the object includes a vehicle and the costs include materials, parts, and labour costs.

[0011] In some embodiments, the expected costs are based in part on historic data of costs of one or more similar impacted components or one or more similar repair operations.

[0012] In some embodiments, the one or more similar impacted components or the one or more similar repair operations are identified with fuzzy matching.

[0013] In some embodiments, the expected costs are based in part on expected one or more repair operations for the one or more impacted components.

[0014] In some embodiments, the expected costs are based in part on defects determined in captured images of the object.

[0015] In some embodiments, the defects are determined by processing the captured images of the object using a cage for image alignment, the cage defining segments of the object, aligning the captured images of the object onto the cage to identify segments of the object in the captured images, and detecting one or more defects in the segments of the object in the captured images.

[0016] In some embodiments, the cage is selected from a plurality ofcages based on at least one of an object identification number, user selection, and system selection.

[0017] In some embodiments, a visual language extractor is used to process the representation of the invoice.

[0018] In some embodiments, the visual language extractor includes a visual language model.

[0019] In some embodiments, the representation of the invoice are further processed using optical character recognition (OCR) and the system further compares output from the visual language extractor and output from the OCR.

[0020] In some embodiments, the report identifies anomalies between the costs and the expected costs.

[0021] In some embodiments, transmitting or storing the report comprises automatically submitting an estimate.

[0022] In some embodiments, the system further includes a user device with an imager to capture the representation of the invoice.

[0023] In some embodiments, the mapping engine uses Categorization of Component and Operational Costs to map the costs.

[0024] In some embodiments, processing the representation of the invoice to extract costs includes localizing regions of the representation. Regions include a main table and a summary table.

[0025] According to an aspect, there is provided a method for automatically reviewing an invoice for repairing an object. The method including receiving a representation of the invoice, processing the representation of the invoice to extract costs, mapping the costs onto at least one of one or more impacted components and one or more repair operations, comparing the costs to expected costs based on the at least one of one or more impacted components and the one or more repair operations, and transmitting or storing a report on the comparison.

[0026] In some embodiments, the object includes a vehicle and the costs include materials, parts, and labour costs.

[0027] In some embodiments, the expected costs are based in part on historic data of costs of one or more similar impacted components or one or more similar repair operations.

[0028] In some embodiments, the method further includes identifying the one or more similar impacted components or the one or more similar repair operations with fuzzy matching.

[0029] In some embodiments, the expected costs are based in part on expected one or more repair operations for the one or more impacted components.

[0030] In some embodiments, the expected costs are based in part on defects determined in captured images of the object.

[0031] In some embodiments, the defects are determined by processing the captured images of the object using a cage for image alignment, the cage defining segments of the object, aligning the captured images of the object onto the cage to identify segments of the object in the captured images, and detecting one or more defects in the segments of the object in the captured images.

[0032] In some embodiments, the cage is selected from a plurality ofcages based on at least one of an object identification number, user selection, and system selection.

[0033] In some embodiments, a visual language extractor is used to process the representation of the invoice.

[0034] In some embodiments, the visual language extractor includes a visual language model.

[0035] In some embodiments, the representation of the invoice are further processed using OCR and the system further compares output from the visual language extractor and output from the OCR.

[0036] In some embodiments, the report identifies anomalies between the costs and the expected costs.

[0037] In some embodiments, transmitting or storing the report includes automatically submitting an estimate.

[0038] In some embodiments, the method further includes capturing the representation of the invoice.

[0039] In some embodiments, mapping the costs uses Categorization of Component and Operational Costs.

[0040] In some embodiments, processing the representation of the invoice to extract costs includes localizing regions of the representation. Regions include a main table and a summary table.

[0041] According to an aspect, there is provided a system for automatically reviewing an invoice for repairing an object. The system includes a user device including an imager and a server having non-transitory computer readable storage in communication with the user device over a network. The non-transitory computer readable storage storing executable instructions for causing one or more processors to receive a captured image of the invoice from the user device, the image captured using the imager, process the captured image of the invoice using a visual language extractor comprising a visual language model to extract costs by localizing regions including a main table and a summary table in the captured invoice, map the costs onto at leastone of one or more impacted components and one or more repair operations (including attaching costs in the summary table to the main table) using a mapping engine, compare the costs to expected costs based on the at least one of the one or more impacted components and the one or more repair operations, and transmit or store a report on the comparison.

[0042] In some embodiments, the executable instructions further cause the one or more processors to receive captured images of the object from the user device, the images captured using the imager, processing the captured images of the object using a cage for image alignment, the cage defining segments of the object, aligning the captured images of the object onto the cage to identify segments of the object in the captured images, detecting one or more defects in the segments of the object in the captured images, and determining the expected costs based in part on the detected one or more defects.

[0043] In some embodiments, mapping the costs includes detecting duplicate repair operations and including the detected duplicate repair operations in the report on the comparison.

[0044] According to an aspect, there is provided a method of providing a reliability pipeline to machine learning predictions. The method including processing data in redundant steps to predict a result, determining a likelihood estimate that the prediction is wrong, and flagging the prediction for review if the likelihood is above a threshold.

[0045] In some embodiments, processing data in redundant steps includes processing different modes of the data to predict two or more results and the likelihood estimate is based on a level of difference between each of the two or more results.

[0046] In some embodiments, the different modes of data include at least one of visual data and textual data, a visual language extractor output and a character recognition engine output, and a visual invoice mapping output and a defect detection output.

[0047] According to an aspect there is provided a non-transitory computer recordable storage medium having stored therein computer executable program code, which when executed by a processor, causes the processor to carry out methods described herein.DESCRIPTION OF THE FIGURES

[0048] In the figures, embodiments are illustrated by way of example. It is to be expressly understood that the description and figures are only for the purpose of illustration and as an aid to understanding.

[0049] Embodiments will now be described, by way of example only, with reference to the attached figures, wherein in the figures:

[0050] FIG. 1 illustrates a block schematic diagram of an example system for invoice analysis, according to some embodiments.

[0051] FIG. 2 illustrates a process diagram of a method of automatically reviewing an invoice for repairing an object, according to some embodiments.

[0052] FIG. 3 illustrates a process diagram of a method of providing a reliability pipeline to machine learning predictions, according to some embodiments.

[0053] FIG. 4 illustrates a schematic diagram of computing device, according to some embodiments.DETAILED DESCRIPTION

[0054] Described herein are structured extraction methodologies called Visual Invoice Mapping (VIM) to analyze invoices for object repair (e.g., automotive body shop invoices for vehicle repair). VIM can leverage automated tools, such as Visual Language Extractors, Visual Language Models (VLMs), and other Machine Learning methods to iteratively validate and enhance data extraction. The systems and methods described herein can further reconcile the invoice analysis with an assessment of the object (e.g., visually detecting defects in the object using, for example, computer vision). This approach can capture detailed component-level and operational cost data from real-world body shops with, for example, vehicles that have been inspected, thereby potentially providing a robust dataset that can closely algin the assessment estimates with the invoice estimates. With increased frequency of inspections, this can ensure temporal alignment between inspection results and actual repair costs. The systems and methods described herein can provide an accurate fiscal reference for damages identified thereby giving a higher levels of insight to manage fleet repair expenditures.

[0055] VIM can provide a structured approach to transforming and analyzing automotive repair invoices. The systems and methods may address critical challenges in fleet repair management by combining automated data extraction, intelligent categorization, and advanced analytics. The systems and methods described herein may create actionable insights for variance analysis, cost prediction, and anomaly detection in addition to automating the processing of multilingual invoices.

[0056] Example System Implementation.

[0057] FIG. 1 illustrates a block schematic diagram of an example system 100 for invoice analysis, according to some embodiments.

[0058] The system 100 can be used to extract key data points from an invoice captured by a user 10 using a visual language extractor 120. The key data points may then be mapped onto repair operations to be conducted on the object using a mapping engine 122. The mapped data can be used for a variety of purposes. For example, the mapped data can be used to assess whether the invoice is reasonable (e.g., by comparing it to a database of other invoices or comparing it to an assessment of the object). Mapped data may be visually presented to the user 10 with damaged components and / or repair operations displayed on the invoice (showing which invoice line items relate to which underlying components or repair operations). The mapped data may also be provided in a dataset to make more robust models. In some optional embodiments, the system 100 may be configured to assess the object using captured images of the object.

[0059] The system 100 can include an I / O unit 102, a processor 104, a communication interface 106, and data storage 108. The processor 104 can execute instructions in memory 110 to implement aspects of processes described herein. The processor 104 can execute instructions in memory 110 to configure a visual language model 120, a mapping engine 122, an application programming interface (API) 124, and other functions described herein. The system 100 may be software (e.g., code segments compiled into machine code), hardware, embedded firmware, or a combination of software and hardware, according to various embodiments. The system 100 may further comprise other components within its data storage 108 such as databases 112, to provide stored data to memory 110, and persistent storage 114.

[0060] The I / O unit 102 can enable the system 100 to interconnect with one or more input devices, such as a keyboard, mouse, camera, touch screen and a microphone, and / or with one or more output devices such as a display screen and a speaker.

[0061] The processor 104 can be, for example, any type of general-purpose microprocessor or microcontroller, a digital signal processing (DSP) processor, an integrated circuit, a field programmable gate array (FPGA), a reconfigurable processor, or any combination thereof.

[0062] The communication interface 106 can enable the system 100 to communicate with other components, to exchange data with other components, to access and connect to network resources, to serve applications, and perform other computing applications by connecting to a network 140 (or multiple networks) capable of carrying data including the Internet, Ethernet, plain old telephone service (POTS) line, public switch telephone network (PSTN), integrated services digital network (ISDN), digital subscriber line (DSL), coaxial cable, fiber optics, satellite, mobile, wireless (e.g., Wi-Fi, WiMAX), SS7 signaling network, fixed line, local area network, wide area network, and others, including any combination of these.

[0063] The data storage 108 may be configured to store information associated with or created by the system 100, such as for example image data, metadata, object metrics, cost data, defect assessment reports and so on. Data storage device 108 can include memory 110, databases 112, and persistent storage 114. The data storage 108 can implement databases 112, for example. Storage 108 and / or persistent storage 114 may be provided using various types of storage technologies, such as solid state drives, hard disk drives, flash memory, and may be stored in various formats, such as relational databases, non-relational databases, flat files, spreadsheets, extended markup files, and so on.

[0064] Memory 110 may include a suitable combination of any type of computer memory that is located either internally or externally such as, for example, random-access memory (RAM), read-only memory (ROM), compact disc read-only memory (CDROM), electro-optical memory, magneto-optical memory, erasable programmable read-only memory (EPROM), and electrically-erasable programmable read-only memory (EEPROM), Ferroelectric RAM (FRAM) or the like.

[0065] The system 100 can have a visual language extractor 120 to extract key data points from a representation of the invoice (e.g., data files or captured images of the invoice). The visual language extractor 120 can be trained to extract key data points such as parts, labour, and financial details from captured images of repair invoices. A visual language extractor 120 may be better able to understand the context provided in the invoice. For example, a visual language extractor 120 may be better able to understand how table headings relate to the items in the table. Accordingly, visual language extractor 120 may be better able to extract key data points from theinvoice. The automatic extraction of data from differing invoices can streamline the repair order process for fleets of vehicles (e.g., the drivers need only to capture images of the invoice). The system 100 can extract usable and consistent data from a variety of different repair shops that may report the same repairs differently (e.g., some repair shops may provide more detailed breakdowns than others). The system 100 may overcome limitations in existing technologies that can result in inconsistent results, hallucinated outputs, and unknown or invalid predictions. These problems can arise in part in invoices in particular because of complexities such as multi-page tables and diverse table formats.

[0066] In some embodiments, the visual language extractor 120 may include a Visual Language Model. The Visual Language Model may be able to take visual inputs and output a context sensitive string (or other textual output). In some embodiments, the output may be a JSON output.

[0067] In some embodiments, the visual language extractor 120 may be configured to localize the invoice using computer vision by identifying different regions of the document that characteristically contain different types of information. Localizing the information in this way may better process the information in the regions as it may process the same information differently depending on the region it’s found in (e.g., finding the word “Tire” in the header where the name of the company ought to be might suggest something different than finding it in the main table). In some embodiments, these regions may include the header (which may include, for example, metadata such as the vehicle type, the body shop name, etc.), the main table (which may include, for example, a set of operations, unit prices, labour prices, etc.), and a summary table (which may include, for example, the total expected cost). Other types of region are possible. Localization may include, for example, applying a bounding box to the captured image representing the different regions. After localizing the regions, the visual language extractor 120 may perform character recognition using, for example, an OCR on the individual regions to convert the visual input into strings.

[0068] The system 100 can also have a mapping engine 122 to map the parts and labour costs onto the damaged components and types of repair operations performed (e.g., Repair / Replace / Replace & I nstall / Blend etc.). This breakdown can be compared to a database of costs associated with damaged components and types of repair operations performed. This can be used to determine whether the charges are reasonable and report anomalies (e.g., identifywhen the price is unfairly increased or if there are more straightforward repair operations which could be performed).

[0069] The mapping engine 122 may map costs onto the damaged components and / or repair operations. These damaged components and / or repair operations my be determined based on the invoice (e.g., determined based on what the invoice proposes to repair), an expert assessment (e.g., based on a third party analysis of the damaged components and repair operations), or a condition report generated using computer vision (e.g., an automatic damage assessment conducted with computer vision).

[0070] In some embodiments, the mapping engine 122 may take the output from the visual language extractor 120 (e.g., string data and context aware regions) to attach the cost from the summary table to each line item in the main table. The system 100 may be able to look through the invoice to identify duplication of costs. For example, if one repair included the operation of removing the right headlight and another repair included that operation as well, then that operation was likely only carried out once during the actual repair, but it may be double charged if the body shop’s invoicing system is not configured to identify such synergies (referred to as over repairs).

[0071] In some embodiments, the mapping engine 122 may then group the costs based on a hierarchical categorization of costs (e.g., the CANDOC categorization described below). The mapping engine 122 may make use of a large language model as part of the grouping exercise. The large language model may be cued up with the hierarchical categorization system and other information such that the large language model can categorize the information from the invoice into the hierarchical categorization system. The large language model may also predict which components of the vehicle are impacted by the repair operations. Standardizing the costs manner may make it easier for the system 100 to compare costs between different body shops. The system 100 may also take the information from the heading (e.g., state in which the body shop is located) to compare the costs (as this may lead to different standard costs and different labour rates). Comparing the costs can also identify whether there are unusual operations being proposed.

[0072] The system 100 may further use a large language model to take the information from the mapping engine 122 and provide an overall explanation (e.g., in layman’s terms) for the operator or as a summary for other purposes.

[0073] The system 100 may automatically align invoice entries with predefined repair and replacement standards, enabling precise variance analysis. The ability to seamlessly integrate disparate invoice formats into a unified condition report framework, using advanced matching algorithms and categorization schemas, may enhance repair cost management. Mapping of invoice data to, for example, assessment condition reports can bridge the gap between the raw invoices and the standardized condition reports.

[0074] In some embodiments, the mapping engine may use statistical models that can analyze invoice cost distributions to identify similar invoices using fuzzy matching (e.g., scripted invoice similarity matching). This capability can enable targeted sampling of invoice categories for deeper analysis. The fuzzy matching approach to categorizing invoices based on statistical patterns introduces an efficient way to handle diverse and non-standardized data.

[0075] The system 100 can provide a robust and hierarchical taxonomy for categorizing costs at the component and operational levels (otherwise known as Categorization of Component and Operational Costs (CANDOC)). This taxonomy can be used to map invoice data enhancing interpretability. In some embodiments, the system 100 can associate each cost with a component, operation type, and region. These categories can power variance analysis and predictive cost modeling. CANDOC’s ability to categorize costs dynamically, across multiple dimensions, may introduce granularity in repair cost analysis. In some embodiments, it may map the data visually. Combining detailed cost categorization with a visual mapping interface can improve transparency and empower clients to understand and validate repair costs easily.

[0076] The system 100 can provide scripted pipelines for data analysis. This data analysis may aggregate extracted data into meaningful insights which can enable automation of the elements such as body shop address (supports the Geographic Body Labor (GBL) pipeline for regional labor cost analysis), invoice dates (aligns with assessment condition report timelines and supports Visual Invoice Mapping), vehicle identifiers (links specific invoices to corresponding vehicles in the system), main tables of operations (structures body shop operations into standardized tables for categorization in CANDOC), summarized costs (generates aggregated costs for line-item analysis in the cost calibration pipeline). Processing these tasks with the system 100 can reduce human intervention, increase accuracy, and standardize previously unstructured data.

[0077] The system 100 can convert an invoice into damaged components and repair operations and those damaged components and repair operations into cost predictions. These cost predictions can be compared to historic cost predictions to determine the reasonability of the repair costs (e.g., is the cost artificially high or are the repair operations reasonable). The system 100 can extract and evaluate invoice data. In some embodiments, this can convert raw invoice PDFs into structured JSON data for analysis which can ensure accurate and lossless data extraction. A relationship between invoices and assessment condition reports can be established which may validate the extracted data's reliability. For example, each line item can be cross-referenced with the proposed damage or defects (found from the invoice or with an assessment condition report). The system 100 can highlight in-scope, out-of-scope, and anomalous entries, which can create a real-time map of invoice accuracy.

[0078] The system 100 can provide a framework for user-oriented outcomes including actionable insights for users 10, such as variance analysis (e.g., identifying discrepancies between invoice scope and assessment condition report scope using CANDOC), cost differentials (e.g., highlighting part and labor cost variances, enabling more accurate recalibrations), and out-of-scope contributions (e.g., quantifying the cost impact of out-of-scope line items, supporting predictive modeling). The ability to provide users 10 with detailed, component-level cost insights can enhance trust and facilitates informed decision-making.

[0079] In some embodiments, this system 100 may include components for damage assessment using computer vision (e.g., an alignment engine 126 and a recognition engine 128). In some embodiments, the system 100 may make use of cages to detect damage and / or pixelwise comparison. For example, the user 10 may capture images of the object using the image capture module 132 and the system 100 may assess the captured images or the object and predict the damaged components and types of repair operations that may need to be performed to repair the vehicle. For example, the alignment engine 126 may align captured images of an object with, for example, a cage (e.g., a scalable vector cage). Scalable vector cages can be used to recognize object parts accurately, making it particularly useful for situations where objects can be represented as a collection of distinct features. Cages may efficiently segment an object into parts to better identify damage and / or defects and the location of the damage and / or defects. The alignment engine 126 may determine which cage type to use based on the captured images, metadata, inputs from the user, a serial or identification number (e.g., a vehicle identification number or VIN or a license plate), system configuration, or another source. The cage can define segments of an object of that type. The recognition engine 128 may process the captured imagesof the object and / or metadata to compute object metrics. In some embodiments, images may be processed by the recognition engine 128 to identify specific sections and body parts of the object (e.g., the vehicle). The recognition engine 128 may use the aligned cages to identify different locations or components of the object shown in the images. The damage shown in the image can then linked to the respective location or component. These sections can then be inspected by the system 100 to detect any defects or issues. In some embodiments, the system 100 uses the recognition engine 128 for analyzing the captured images of the object to determine, for example, vehicle metrics such as the odometer reading, any detected paint chips, scratches, defects and dents listing the affected body parts and the severity of damage. The recognition engine 128 can be configured to recognize other aspects or features of the parts of an object (e.g., other deviations from a standard or ideal model). The system may use the component and defect location to determine the likely repair operations.

[0080] The system 100 may use outputs from the determination of the damaged components to determine likely repair operations, the cost of said repair operations, and compare those to the extracted data from the invoice. In some embodiments, the system 100 may generate a predicted costforthe damage based on captured images of the object using, for example, a cost estimation tool. The cost estimation tool can process object metrics to detect defects of the object and predict cost data for repair of the defects. These predictions can be compared to the prices extracted from the invoice to determine whether they are reasonable. In such embodiments, the system 100 may be able to identify unnecessary or optional repairs in the invoice or may identify different repair operations to repair the vehicle (e.g., repair a damaged part instead of replacing it).

[0081] Damage detection or other aspects of the system 100 described herein may be carried out using the systems and methods described in US Patent Application No 17 / 286,401 , titled “Automated artifical intelligence vehicle appraisals”, filed on 16 October 2019, the contents of which are incorporated herein by reference. For example, a system for vehicle appraisals (or routine inspection) using image processing may be provided. The system may have server having non-transitory computer readable storage medium with executable instructions for causing one or more processors to configure an interface application with a vehicle capture module to capture images of a vehicle and metadata for the captured images, the interface application displaying an interactive guide to assist in capturing the images, the interactive guide generated using a cage for a vehicle type, the cage defining locations or components of the vehicle, a vehicle identification number being metadata for the captured images, the vehicle identification number indicating the vehicle type, a recognition engine to process the captured images and metadata to detect defectsof the vehicle and compute vehicle metrics, the processing based on different tasks dispatched to agent interfaces to receive input data for detecting the defects of the vehicle and computing the vehicle metrics, a cost estimate tool to process the vehicle metrics to compute cost data for repair of the defects of the vehicle, a valuation tool to compute a market value estimate for the vehicle using the vehicle metrics and the cost data. The interface application has visual elements corresponding to the interactive guide, the market value estimate, the cost data, and at least a portion of the vehicle metrics. For example, a valuation of an object may be carried out in parallel with invoice mapping to consider whether it would be worthwhile to repair the object or to sell it or scrap it for parts (e.g., to wholly replace the object).

[0082] Damage detection or other aspects of the system 100 described herein may be carried out using the systems and methods described in US Patent Application No 18 / 720,315, titled “SCALABLE VECTOR CAGES: VECTOR-TO-PIXEL METADATA TRANSFER FOR OBJECT PART CLASSIFICATION”, filed on 22 March 2024, the contents of which are incorporated herein by reference. For example, a system for classifying segments of an object may be provided. The system can include a server having non-transitory computer readable storage medium with executable instructions. The executable instructions for causing one or more processors to process captured images using a plurality of cages to identify a cage for image alignment, the cage defining segments of the object, align the captured images onto the cage to identify segments of the object in the captured images, and detect one or more physical conditions (e.g., defects) of the segments of the object in the captured images. For example, cages may be used to identify damage and specific damaged components as part of a damage assessment to be compared to the repair operations proposed on the invoice.

[0083] Damage detection or other aspects of the system 100 described herein may be carried out using the systems and methods described in provisional US Patent Application No 63 / 665,992, titled “SCALABLE VECTOR CAGES: VECTOR-TO-PIXEL METADATA TRANSFER FOR DEFECT ALIGNMENT”, filed on 28 June 2024, the contents of which are incorporated herein by reference. For example, a system for classifying segments of an object may be provided. The system can include a server having non-transitory computer readable storage medium with executable instructions for causing one or more processors to process captured images using a cage for image alignment, the cage defining segments of the object, align the captured images onto the cage to identify segments of the object in the captured images, detect one or more defects in the segments of the object in the captured images, append data representative of the one or more defects to the cage. For example, the cages may include defectspreviously identified on the vehicle which may aid in determining which operator damaged the object and may factor into which defects may be covered by, for example, insurance from the repairs on an invoice.

[0084] The system 100 can have an application programming interface (API) 124 to integrate with other systems for data exchange.

[0085] The system 100 can be operable to register and authenticate users (using a login, unique identifier, and password for example) prior to providing access to applications, a local network, network resources, other networks and network security devices. The system 100 can connect to different machines, and / or data sources.

[0086] The system 100 can configure an interface application 130 with an image capture module 132 to capture images of an invoice or of an object and / or metadata for the captured images. The system 100 may configure an interface application 130 with visual elements to guide a user through an image capture process using object capture module 132. In some embodiments, the image capture module 132 may be configured to capture images by pulling frames from a feed of video. In some embodiments, the images may be captured in some other way from the video feed.

[0087] The system 100 may be configured for automatically reviewing an invoice for repairing an object. In some embodiments, the system 100 can be used for detecting damage to an object such as a vehicle. In some embodiments, the system 100 can be used to assist with ongoing maintenance of an object. The system 100 can include data storage 108 having non-transitory computer readable storage 110 with executable instructions. The instructions may be for causing one or more processors 104 to receive a representation of the invoice, process the representation of the invoice to extract costs (using, e.g., a visual language extractor 120), map the costs onto one or more impacted components and one or more repair operations (using, e.g., a mapping engine 122), compare the costs to expected costs based on the one or more impacted components and the one or more repair operations, and transmit a report on the comparison. In some embodiments, the user 10 may capture the representation of the invoice as an image of the invoice using a user device with, for example, an image capture module 132 thereon. The approaches described herein can provide invoice data extraction that may generate an output that accurately captures essential information from, for example, invoice headers, main tables,and financial summaries, which may enable fine-grained comparison of the effective costs associated with damages.

[0088] In some embodiments, optical character recognition (OCR) may also be used to process the invoice. To use OCR to extract key data points, subsequent processing steps may be required. The OCR analysis of the invoice may be able to provide an OCR output that can be compared to a Visual Language Extractor output. While the Visual Language Extractor output may be more a contextual than the OCR output, the OCR output may nonetheless be useful to ensure that the Visual Language Extractor output has captured all relevant data (e.g., by ensuring that some or all information in the OCR output is represented somewhere in the visual language extractor output). This may be useful to provide a reliability pipeline for the visual language extractor 120 as a quality assurance measure.

[0089] In some embodiments, the system 100 may be multi-lingual. For example, the system 100 may be able to match different ways of expressing the same underlying repair operations. Different invoices may communicate the same underlying repair operations differently. By extracting this information and mapping it in a more uniform way, the data may better be able to be compared to other data (e.g., historic data or data from other repair invoices). This can generate a more consistent dataset which may be subsequently used for comparison purposes and / or model training purposes which may render system 100 more efficient. Furthermore, it may be better able to partition the data to flag specific portions of the invoice that may be anomalous (e.g., as opposed to indicated the invoice itself is too expensive, it may be able to indicate that there is an unnecessary repair operation proposed, or the labour rates or part / material costs are too high). This may put the driver and the system 100 in a better position to negotiate prices or propose alternative repair operations.

[0090] In some embodiments, the system 100 may have multi-lingual capabilities to provide accessibility and clarity across global markets. Such embodiments may be capable of articulating damage in multiple languages making it suitable for international deployment and ensuring that results are both accurate and comprehensible to diverse user. This can help ensure consistency across different regions and in different languages.

[0091] In some embodiments, the system 100 may further be configured to receive captured images of the object, detect defects in the captured images of the object, predict the expected cost based on the defects detected in the captured images of the object. In such embodiments,the system 100 may implement various computer vision techniques to process the images and predict the expected costs (e.g., cages, pixel-wise comparison, etc.). For example, the captured images of the object may be processed with, for example, cages (e.g., Scalable Vector Cages (SVC)). Using cages may provide the advantage of being able to segment an image of the object into its component parts which may better aid in ascertaining which components are defective or damaged and what the repair operations should be. For example, the system 100 may be configured to process the captured images of the object using a cage (the cage defining segments of the object, e.g., an SVC) for image alignment by the alignment engine 126, aligning the captured images of the object onto the cage to identify segments of the object in the captured images using the alignment engine 126, and detecting one or more defects in the segments of the object in the captured images using the recognition engine 128. In some embodiments, the system 100 may be configured to select a cage from a plurality of cages based on, for example, an object identification number, user selection, and system selection (e.g., the system may test various cages against the object and select the cage that provides the best match).

[0092] In some embodiments, the system 100 may save the output for future comparisons. For example, this may help build a robust format-agnostic dataset that includes rarer forms of damage for future comparison. In some embodiments, the cost extracted from the invoice may be used as ground truth to train any model that independently assesses the object for defects. These outputs may be used to train future machine learning models that may be used to predict the expected costs. Using a self-supervised learning approach, the system 100 can continuously learn from new invoices which can improve its ability to process previously unseen formats.

[0093] The system 100 may dynamically expand its invoice dataset by incorporating diverse invoices from varying regions, languages, and operational contexts. The system 100 may identify patterns and fill data gaps by utilizing machine learning and statistical techniques which may create a comprehensive repository for cost estimation. The capacity of the system 100 to ingest multilingual invoices and adapt to new formats without requiring pre-defined templates introduces a scalable and adaptive solution to invoice processing.

[0094] Some embodiments may provide the advantage of identifying differences between repair shop invoices and cost estimates from object assessments (e.g., machine vision defect detection). Such differences may arise from missed defects in the object assessment (e.g., repairs on the roof of a vehicle or the interior) and variances due to vehicle type, labour rate differentials,and random variations related to the cost of necessary operations to repair damages or replace components based on damage severity.

[0095] The system 100 may use the outputs, for example, to train or refine defect detection models that use computer vision to detect defects in the object. The invoices (subject to data processing, e.g., to remove outliers) may be used as ground truth for computer vision defect detection. This may enable the models to more readily identify damage that is not apparent in the images (e.g., damages in hard to see areas such as the roof or the interior). This may make a model that is also able to account for regional differences in cost estimation.

[0096] The invoice may generally refer to a list of goods and / or services with a sum due for these good and services. The invoice may include an itemized list with sums due for each good or service provided. Some invoices may include more or less information than another invoices for the same work (e.g., one invoice may provide a detailed and itemized breakdown of costs, others may merely provide a total sum owed). In some embodiments, the invoice may generally describe the costs associated with repair operations performed on an object to repair damaged and / or defective components of the object. In some embodiments, the extracted costs may include costs based on the hours of labour and the hourly rate of said labour, parts, materials, or other metrics. The visual language extractor 120 may be able to extract from the invoices a consistent format of repair operations and the mapping engine 122 may be able to associate the costs therewith.

[0097] The representation of the invoice can be, for example, a captured image of the invoice. The representation of the invoice can be, for example, a virtual copy of an invoice (e.g., a word document, portable document formal file (PDF), etc.). In some embodiments, the representation of the invoice may include metadata associated with the invoice (e.g., time of image, time of repairs, etc.).

[0098] The repair operations may refer to operations that can be conducted indented to bring a damaged and / or defective component withing operational tolerances for use. In some embodiments, repair operations can include repairing a damaged part, replacing the damaged part, removal and installation of a part (e.g., to get at a damaged part underneath it), etc. To repair damaged or defective components, some components that are not defective or damaged may be impacted (e.g., to get access to the part in question) which may give rise to costs associated therewith.

[0099] In some embodiments, the expected costs can be based on historic data such as historic costs for the same repair operations on the same impacted components. In some embodiments, the historic data may be for similar repair operations for similar impacted components. For example, the system 100 may base an expected cost on the same repair operations on the same components, but in a different type of object (e.g., type of vehicle). The system 100 may use similar repair operations to repair the same impacted component. The system 100 may use the same repair operations to repair a similar component. In some embodiments, the system 100 can use the best match (e.g., object type, component, and repair operation) that it can for the historic data.

[0100] In some embodiments, the expected costs can be based on the expected repair operations. For example, the system 100 may be configured to identify (e.g., based on the invoice, a damage assessment conducted by an expert, or a damaged assessment conducted using computer vision) an expected one or more repair operations. The system 100 may flag when there is a difference between the actual invoice costs and the expected costs because one or more unexpected repair operations are proposed to be undertaken (e.g., where the component could have been repaired, but the invoice is for a replacement).

[0101] In some embodiments, the output of the system 100 can identify anomalies between the cost of the invoice and the expected costs. This can flag that there may be an issue with the invoice. In some embodiments, the report can identify why an anomaly has been flagged. For example, the system 100 may identify that one has been flagged because the cost is far higher than the expected cost (e.g., above an absolute and / or relative threshold) or the system 100 may identify that unnecessary or different repair operations exist within the invoice than were predicted to repair the object based on the damage or defects.

[0102] In some embodiments, the system 100 may output the estimate for a repair claim automatically. This may ensure that the actual estimate provided on the invoice is included in the claim. It may also reduce the likelihood of fraud or error by those inputting the invoice information into any claim system. This may also input a timestamp of the invoice.

[0103] Fraudulent repair claims often involving inflated invoices. These schemes may rely on discrepancies between actual repair costs and the submitted invoice amounts, enabling individuals to pocket the difference. Indicators can include backdated invoices, misaligned timestamps between condition reports and invoices, and unexplained high costs, all of which may becaptured by the solution described herein. Timestamps may be automatically validated based on digital records that capture the vehicle condition before it goes to the body-shop. The resulting invoices from the body-shop may highlight repair completion dates along with detailed descriptions of activities which can be requested and cross-referenced with condition reports. In this way the solution described herein can offer a cost-effective way to monitor the chain of custody. Body-shop documents can be uploaded to the system to monitor unexpected repairs and to flag anomalies in repair timing and discrepancies with estimated labor and material costs thereby preventing fraudulent activities.

[0104] The system 100 can perform refined repair invoice analysis and in so doing may provide enhanced repair estimates. Inconsistencies as between an assessment estimate and the invoice estimate can then be highlighted for the user. The system 100 may extract structured data from the invoices. This extracted data may provide ground truth data that can better reveal the financial impact of damages identified by defect detection systems. The system 100 may better enable users 10 to track, predict, and manage repair costs, and enhance transparency and operational efficiency across, for example, a fleet of vehicles. The system 100 may offer accurate fiscal references for damages identified and providing insight to manage fleet repair expenditures.

[0105] The system 100 can help establish a robust data extraction and analysis framework that can accurately capture discrepancies between Fleet Management Companies (FMCs) repair invoices and assessment estimates. Misalignment between the invoices and the assessment estimates may arise from hidden repair costs for undetected damages (e.g., roof and interior repairs). The extracted data can serve as a foundation for refining the assessment estimate model which can enable more precise and region-specific cost predictions. With the use of this data, model outputs may be more closely aligned with actual repair outcomes which can enhance predictive accuracy and provide actionable insights for continuous improvement.

[0106] In some embodiments, the system 100 can combine structured data and cost categorization into an interactive visual interface. Users 10 can trace costs back to specific invoice sections, verify scope, and analyze part and labor differentials. The system 100 can improve usability and transparency by integrating data visualization directly into the analysis pipeline.

[0107] The systems 100 may provide a comprehensive solution to relieve these financial strains. The systems and methods described herein can combine a highly precise and finegrained condition monitoring solution with damage repair estimates. Condition monitoring by thesystem 100 can be conducted by anyone, anywhere if there’s enough room to walk around the vehicle and take pictures with a cell-phone. The system 100 can track the resulting imagery over time using machine learning models that can produce highly detailed condition reports which can comply with industry standards for expert inspections. The outputs can comply with these standards using a Human-in-the-Loop design. The condition of the vehicles can be estimated in terms of the type and severity of damages, and the appropriate costs of repairing or replacing affected components at a body-shop. The costs can be estimated by the system 100, which can process large collections of body-shop invoices, thereby allowing FMCs to continually calibrate their strategies with real-time information. Users may apply these approaches to eliminate financial risks that reduce profitability such as extended downtime, over-repair, fraud and underutilized insurance recoveries.

[0108] The system 100 can mitigate increasing costs by capturing and aligning different types of data sources using advanced machine learning models. The information that is produced in turn can enhance transparency and offers FMCs actionable insights to advance opportunities for cost savings and to alleviate fraud.

[0109] With the use of automated procedures for monitoring the vehicle condition and bodyshop operations, digital condition records and data-driven cost estimates can be integrated in a centralized data repository for fleet managers. The herein described data-driven approach can facilitate transparency in decision-making, and proactive cost management, which in turn can enable FMCs to leverage the main benefit of Al through continuously improving decision making.

[0110] The system 100 can support improving decision-making by integrating digital condition records with data-driven cost estimates into a centralized data repository for fleet managers. This can enable a cost-effective chain of custody for vehicles in large fleets. With this centralized source of information, the system 100 can produce accurate, data-driven cost estimates that can be used to proactively procure body-shop activities. It can also facilitate retroactive validation of invoices to ensure that FMCs are less likely to fall victim to fraudulent claims and inflated costs. This solution can provide cost savings and bolstered corporate accountability, making it an invaluable tool for FMCs.

[0111] According to an aspect, there is provided a system 100 for automatically reviewing an invoice for repairing an object. The system 100 includes a server having non-transitory computer readable storage 110 with executable instructions for causing one or more processors 104 toreceive a representation of the invoice, process the representation of the invoice to extract costs, map the costs onto at least one of one or more impacted components and one or more repair operations using a mapping engine using a mapping engine 122, compare the costs to expected costs based on the at least one of the one or more impacted components and the one or more repair operations, and transmit or store a report on the comparison.

[0112] In some embodiments, the object includes a vehicle and the costs include materials, parts, and labour costs.

[0113] In some embodiments, the expected costs are based in part on historic data of costs of one or more similar impacted components or one or more similar repair operations.

[0114] In some embodiments, the one or more similar impacted components or the one or more similar repair operations are identified with fuzzy matching.

[0115] In some embodiments, the expected costs are based in part on expected one or more repair operations for the one or more impacted components.

[0116] In some embodiments, the expected costs are based in part on defects determined in captured images of the object.

[0117] In some embodiments, the defects are determined by processing the captured images of the object using a cage for image alignment, the cage defining segments of the object, aligning the captured images of the object onto the cage to identify segments of the object in the captured images using the alignment engine 126, and detecting one or more defects in the segments of the object in the captured images using the recognition engine 128.

[0118] In some embodiments, the cage is selected from a plurality of cages based on at least one of an object identification number, user selection, and system selection.

[0119] In some embodiments, a visual language extractor 120 is used to process the representation of the invoice.

[0120] In some embodiments, the visual language extractor 120 includes a visual language model.

[0121] In some embodiments, the representation of the invoice is further processed using optical character recognition (OCR) and the system further compares output from the visual language extractor and output from the OCR.

[0122] In some embodiments, the report identifies anomalies between the costs and the expected costs.

[0123] In some embodiments, transmitting or storing the report comprises automatically submitting an estimate.

[0124] In some embodiments, the system further includes a user device with an imager to capture the representation of the invoice.

[0125] In some embodiments, the mapping engine 122 uses Categorization of Component and Operational Costs to map the costs.

[0126] In some embodiments, processing the representation of the invoice to extract costs includes localizing regions of the representation. Regions include a main table and a summary table.

[0127] According to an aspect, there is provided a system 100 for automatically reviewing an invoice for repairing an object. The system 100 includes a user device including an imager and a server having non-transitory computer readable storage 110 in communication with the user device over a network. The non-transitory computer readable storage 110 storing executable instructions for causing one or more processors 104 to receive a captured image of the invoice from the user device, the image captured using the imager, process the captured image of the invoice using a visual language extractor 120 comprising a visual language model to extract costs by localizing regions including a main table and a summary table in the captured invoice, map the costs onto at least one of one or more impacted components and one or more repair operations (including attaching costs in the summary table to the main table) using a mapping engine 122, compare the costs to expected costs based on the at least one of the one or more impacted components and the one or more repair operations, and transmit or store a report on the comparison.

[0128] In some embodiments, the executable instructions further cause the one or more processors 104 to receive captured images of the object from the user device, the images captured using the imager, processing the captured images of the object using a cage for imagealignment, the cage defining segments of the object, aligning the captured images of the object onto the cage to identify segments of the object in the captured images using the alignment engine 126, detecting one or more defects in the segments of the object in the captured images using the recognition engine 128, and determining the expected costs based in part on the detected one or more defects.

[0129] In some embodiments, mapping the costs includes detecting duplicate repair operations and including the detected duplicate repair operations in the report on the comparison.

[0130] Example Method.

[0131] FIG. 2 illustrates a process diagram of a method 200 of automatically reviewing an invoice for repairing an object, according to some embodiments.

[0132] The method 200 can include the steps of receiving a representation of the invoice (block 202), processing the representation of the invoice to extract costs (block 204), mapping the costs onto one or more impacted components and one or more repair operations (block 206), comparing the costs to expected costs based on the one or more impacted components and the one or more repair operations (block 208), and transmitting a report on the comparison (block 210).

[0133] Some example methods 200 may further include steps of receiving captured images of the object, detecting defects in the captured images of the object, predicting the expected cost based on the defects detected in the captured images of the object.

[0134] According to an aspect, there is provided a method 200 for automatically reviewing an invoice for repairing an object. The method 200 including receiving a representation of the invoice (block 202), processing the representation of the invoice to extract costs (block 204), mapping the costs onto at least one of one or more impacted components and one or more repair operations (block 206), comparing the costs to expected costs based on the at least one of one or more impacted components and the one or more repair operations (block 208), and transmitting or storing a report on the comparison (block 210).

[0135] In some embodiments, the object includes a vehicle and the costs include materials, parts, and labour costs.

[0136] In some embodiments, the expected costs are based in part on historic data of costs of one or more similar impacted components or one or more similar repair operations.

[0137] In some embodiments, the method 200 further includes identifying the one or more similar impacted components or the one or more similar repair operations with fuzzy matching.

[0138] In some embodiments, the expected costs are based in part on expected one or more repair operations for the one or more impacted components.

[0139] In some embodiments, the expected costs are based in part on defects determined in captured images of the object.

[0140] In some embodiments, the defects are determined by processing the captured images of the object using a cage for image alignment, the cage defining segments of the object, aligning the captured images of the object onto the cage to identify segments of the object in the captured images, and detecting one or more defects in the segments of the object in the captured images.

[0141] In some embodiments, the cage is selected from a plurality ofcages based on at least one of an object identification number, user selection, and system selection.

[0142] In some embodiments, a visual language extractor is used to process the representation of the invoice.

[0143] In some embodiments, the visual language extractor includes a visual language model.

[0144] In some embodiments, the representation of the invoice are further processed using OCR and the system further compares output from the visual language extractor and output from the OCR.

[0145] In some embodiments, the report identifies anomalies between the costs and the expected costs.

[0146] In some embodiments, transmitting or storing the report (block 210) includes automatically submitting an estimate.

[0147] In some embodiments, the method 200 further includes capturing the representation of the invoice.

[0148] In some embodiments, mapping the costs uses Categorization of Component and Operational Costs.

[0149] In some embodiments, processing the representation of the invoice to extract costs includes localizing regions of the representation. Regions include a main table and a summary table.

[0150] Quality Assurance.

[0151] The systems and methods described herein can produce reliable and consistent results by overcoming technical challenges such as variable lighting, angles, and environmental conditions. Such challenges can create inconsistencies in visual data which can affect inspection accuracy. The technical challenges may be addressed through advanced computer vision techniques that can standardize image processing and minimize the influence of external factors. This can aide the machine learning models to consistently recognize and evaluate damage across varied conditions which can be important for remote inspections and mobile inspections where environmental control is limited.

[0152] In some embodiments, the systems and methods described herein may make use of multi-modal architecture to assist with reliability. Such architectures may integrate visual, textual, and tabular sources to create a redundancy protocol that combines the benefits of each mode of input. Such multi-modal capabilities can be leveraged with probabilistic modeling to deliver confidence scores with each prediction which can guide the decision-making process. By incorporating probability-based insights, the systems and methods described herein can automatically handle high-certainty cases while flagging low-confidence assessments for human review, ensuring that each decision can be both precise and well-substantiated.

[0153] For example, the system may make use of different modes of data that represent the same information. Accordingly, the system may be used to compare the information retrieved from the invoice with information retrieved from a visual inspection of the vehicle. Differences between these two modes of data processing may signify an issue with one of these modes which may merit review by a human. If, after human review, the inconsistency still exists, then the system may identify an anomaly with the invoice (e.g., overcharging or anomalous repair operation proposals).

[0154] The systems and methods described herein may be delivered by way of an API-first machine learning solution that can present an approach to affordably handle large volumes of inspections whilst meeting all the different certified standards associated with use cases like retail, reconditioning, and fleet management. The approach described herein may reduce human error and minimize data inconsistencies with vehicle inspections, while delivering results in timeframes unattainable through manual methods.

[0155] The systems described herein may provide efficiency, transparency, and reliability. The approach can resolve edge cases and particularly challenging inspections with the aid of human ingenuity while minimizing the risk of Al errors. The machine learning pipeline can process the data in stages that consist of redundant steps which can leverage the benefits of different machine learning models. These machine learning models can be multi-modal and trained on large, high quality datasets not only for analysis but also for quality control and quality assurance. This can ensure that the platform produces reliable results at scale, while enabling vehicle owners to capture and securely manage their own inspection data. Through a combination of these advanced algorithms and human oversight, the system can provide inspections that are highly accurate and consistent at scale.

[0156] The approach described herein can be highly adaptable to different systems for data management. This integration can ensure that inspection data can flow in real-time across various departments (e.g., sales, service, and other operations) to provide immediate access to actionable insights. This approach can help organizations streamline operations, reduce latency, and improve decision-making, which may be useful for managing high-volume inspections in today’s fast-paced environment.

[0157] The methods may make use of machine learning to make predictions that are associated with a likelihood estimate for how wrong a given result might be. The likelihood estimate can be based on, for example, differences between different modes of redundant data processing. This can help address data quality and algorithmic challenges with advanced computer vision techniques that standardize image processing regardless of image resolution. This may reduce the influence of external factors such as lighting and camera perspectives. Additionally, different machine learning methods may each have a role in a pipeline consisting of separate processing steps for quality control, analysis, and quality assurance of results.

[0158] By training machine learning models to perform quality control, some embodiments of the systems and methods described herein can guide the user to capture photographs that are of sufficient quality for a fine-grained assessment of damages or for better invoice analysis. Data analysis can be performed to privatize uploaded data, recognize identifiable information such as VIN, mileage, license plates and other serial codes as well as cosmetic defects and vehicle components. Furthermore, multi-modal machine learning algorithms can be trained to provide explanations for quality assurance of any generated results. The results of this redundant approach can further be reinforced with human intervention. For example, human intervention may be strategically enforced based on the collection of likelihood estimates that indicate the success of individual machine learning methods. This ensures that high-certainty results can be automatically handled while other results can be flagged for human review for well-substantiated precision enhancements. This human oversight can enable the system to maintain a level of adaptability that can help handle complex or ambiguous cases as well as providing feedback for continued improvements to the machine learning models.

[0159] As another example, a comparison of the invoice processed using the visual language extractor 120 with a straight OCR may identify errors. For example, while the OCR may be unable to identify context of the invoice (e.g., the regions in which the text is presented), the information recognized by the OCR should nonetheless all (or mostly) be accounted for in the output of the visual language extractor 120. Accordingly, running the visual language extractor 120 in parallel with an OCR may provide a means of offering quality assurance.

[0160] The types of machine learning methods used may be based on deep learning methods. These methods may continually adapt to new data streams after being trained on large datasets. These resulting methods can generally be known as Foundation Models. These techniques can be associated with multi-modal GenAI models that can process textual data across different languages. These multi-modal capabilities can provide access to a multilingual solution which can enhances usability for diverse users across a wide range of use cases and geographic locations, to articulate specific damages and required repairs. This can improve consistency in inspection standards across different regions.

[0161] The damage detection and classification may provide a granular and customizable digital system for recognition of cosmetic defects.

[0162] FIG. 3 illustrates a process diagram of a method 300 of providing a reliability pipeline to machine learning predictions, according to some embodiments.

[0163] According to an aspect, there is provided a method 300 of providing a reliability pipeline to machine learning predictions. The method 300 including processing data in redundant steps to predict a result (block 302), determining a likelihood estimate that the prediction is wrong (block 304), and flagging the prediction for review if the likelihood is above a threshold (block 306).

[0164] In some embodiments, processing data in redundant steps (block 302) includes processing different modes of the data to predict two or more results and the likelihood estimate is based on a level of difference between each of the two or more results.

[0165] In some embodiments, the different modes of data include at least one of visual data and textual data, a visual language extractor output and a character recognition engine output, and a visual invoice mapping output and a defect detection output.

[0166] Computing Device Implementation Details.

[0167] The embodiments of the devices, systems and methods described herein may be implemented in a combination of both hardware and software. These embodiments may be implemented on programmable computers, each computer including at least one processor, a data storage system (including volatile memory or non-volatile memory or other data storage elements or a combination thereof), and at least one communication interface.

[0168] Program code is applied to input data to perform the functions described herein and to generate output information. The output information is applied to one or more output devices. In some embodiments, the communication interface may be a network communication interface. In embodiments in which elements may be combined, the communication interface may be a software communication interface, such as those for inter-process communication. In still other embodiments, there may be a combination of communication interfaces implemented as hardware, software, and combination thereof.

[0169] Throughout this discussion, numerous references have been made regarding servers, services, interfaces, portals, platforms, or other systems formed from computing devices. It should be appreciated that the use of such terms is deemed to represent one or more computing devices having at least one processor configured to execute software instructions stored on a computerreadable tangible, non-transitory medium. For example, a server can include one or more computers operating as a web server, database server, or other type of computer server in a manner to fulfill described roles, responsibilities, or functions.

[0170] One should appreciate that the systems and methods described herein may provide better memory usage, improved processing, improved bandwidth usage, efficient image processing, accurate defect identification, enhanced cost estimates, etc.

[0171] The technical solution of embodiments may be in the form of a software product. The software product may be stored in a non-volatile or non-transitory storage medium, which can be a compact disk read-only memory (CD-ROM), a USB flash disk, or a removable hard disk. The software product includes a number of instructions that enable a computer device (personal computer, server, or network device) to execute the methods provided by the embodiments.

[0172] The embodiments described herein can be implemented by physical computer hardware, including computing devices, servers, receivers, transmitters, processors, memory, displays, and networks. The embodiments described herein provide useful physical machines and particularly configured computer hardware arrangements. The embodiments described herein are directed to electronic machines and methods implemented by electronic machines adapted for processing and transforming electromagnetic signals which represent various types of information. The embodiments described herein pervasively and integrally relate to machines, and their uses; and the embodiments described herein have no meaning or practical applicability outside their use with computer hardware, machines, and various hardware components. Substituting the physical hardware particularly configured to implement various acts for nonphysical hardware, using mental steps for example, may substantially affect the way the embodiments work. Such computer hardware limitations are clearly essential elements of the embodiments described herein, and they cannot be omitted or substituted for mental means without having a material effect on the operation and structure of the embodiments described herein. The computer hardware is essential to implement the various embodiments described herein and is not merely used to perform steps expeditiously and in an efficient manner.

[0173] FIG. 4 illustrates a schematic diagram of computing device 400, according to some embodiments.

[0174] For simplicity only one computing device 400 is shown but system may include more computing devices 400 operable by users to access remote network resources and exchangedata. The computing devices 400 may be the same or different types of devices. As depicted, computing device 400 includes at least one processor 402, memory 404, at least one I / O interface 406, and at least one network interface 408. The computing device components may be connected in various ways including directly coupled, indirectly coupled via a network, and distributed over a wide geographic area and connected via a network (which may be referred to as “cloud computing”). For example, and without limitation, the computing device 400 may be a server, network appliance, set-top box, embedded device, computer expansion module, personal computer, laptop, personal data assistant, cellular telephone, smartphone device, UMPC tablets, video display terminal, gaming console, electronic reading device, and wireless hypermedia device or any other computing device capable of being configured to carry out the methods described herein.

[0175] Each processor 402 may be, for example, any type of general-purpose microprocessor or microcontroller, a digital signal processing (DSP) processor, an integrated circuit, a field programmable gate array (FPGA), a reconfigurable processor, a programmable read-only memory (PROM), a graphics processing unit (GPU), or any combination thereof.

[0176] Memory 404 may include a suitable combination of any type of computer memory that is located either internally or externally such as, for example, random-access memory (RAM), read-only memory (ROM), compact disc read-only memory (CDROM), electro-optical memory, magneto-optical memory, erasable programmable read-only memory (EPROM), and electrically-erasable programmable read-only memory (EEPROM), Ferroelectric RAM (FRAM) or the like.

[0177] Each I / O interface 406 enables computing device 400 to interconnect with one or more input devices, such as a keyboard, mouse, camera, touch screen and a microphone, or with one or more output devices such as a display screen and a speaker.

[0178] Each network interface 408 enables computing device 400 to communicate with other components, to exchange data with other components, to access and connect to network resources, to serve applications, and perform other computing applications by connecting to a network (or multiple networks) capable of carrying data including the Internet, Ethernet, plain old telephone service (POTS) line, public switch telephone network (PSTN), integrated services digital network (ISDN), digital subscriber line (DSL), coaxial cable, fiber optics, satellite, mobile, wireless (e.g. Wi-Fi, WiMAX), SS7 signaling network, fixed line, local area network, wide area network, and others, including any combination of these.

[0179] Computing device 400 is operable to register and authenticate users (using a login, unique identifier, and password for example) prior to providing access to applications, a local network, network resources, other networks and network security devices. Computing devices 400 may serve one user or multiple users.

[0180] General Implementation Details

[0181] Applicant notes that the described embodiments and examples are illustrative and non-limiting. Practical implementation of the features may incorporate a combination of some or all of the aspects, and features described herein should not be taken as indications of future or existing product plans. Applicant partakes in both foundational and applied research, and in some cases, the features described are developed on an exploratory basis.

[0182] The foregoing has been described with particular attention to the application to segmenting parts of a vehicle for exemplary purposes only. The systems and methods described herein, even when explicitly referencing a vehicle, are broadly applicable to many object categories. Other applicable object categories include other merchandise, industrial equipment, aircraft parts and components, construction equipment and machinery, medical equipment and devices, electronic devices and components, furniture and fixtures, agricultural machinery and equipment, marine vessels and components, manufacturing machinery and equipment, power generation and distribution equipment, scientific instruments, equipment, and more generally, any item with parts. These processes may also be useful in any application where something needs to be visually inspected, particularly complex objects comprising parts.

[0183] The foregoing discussion provides many example embodiments. Although each embodiment represents a single combination of inventive elements, other examples may include all possible combinations of the disclosed elements. Thus if one embodiment comprises elements A, B, and C, and a second embodiment comprises elements B and D, other remaining combinations of A, B, C, or D, may also be used.

[0184] The term “connected” or “coupled to” may include both direct coupling (in which two elements that are coupled to each other contact each other) and indirect coupling (in which at least one additional element is located between the two elements).

[0185] Although the embodiments have been described in detail, it should be understood that various changes, substitutions and alterations can be made herein without departing from thescope. Moreover, the scope of the present application is not intended to be limited to the particular embodiments of the process, machine, manufacture, composition of matter, means, methods and steps described in the specification.

[0186] Moreover, the scope of the present application is not intended to be limited to the particular embodiments of the process, machine, manufacture, composition of matter, means, methods and steps described in the specification. As one of ordinary skill in the art will readily appreciate from the disclosure, processes, machines, manufacture, compositions of matter, means, methods, or steps, presently existing or later to be developed, that perform substantially the same function or achieve substantially the same result as the corresponding embodiments described herein may be utilized. Accordingly, the appended claims are intended to include within their scope such processes, machines, manufacture, compositions of matter, means, methods, or steps.

[0187] As can be understood, the examples described above and illustrated are intended to be exemplary only. The scope is indicated by the appended claims.

Claims

WHAT IS CLAIMED IS:1 . A system for automatically reviewing an invoice for repairing an object, the system comprising:a server having non-transitory computer readable storage with executable instructions for causing one or more processors to:receive a representation of the invoice;process the representation of the invoice to extract costs;map the costs onto at least one of one or more impacted components and one or more repair operations using a mapping engine;compare the costs to expected costs based on the at least one of the one or more impacted components and the one or more repair operations; and transmit or store a report on the comparison.

2. The system of claim 1 , wherein the object comprises a vehicle and the costs comprise materials, parts, and labour costs.

3. The system of claim 1 or 2, wherein the expected costs are based in part on historic data of costs of one or more similar impacted components or one or more similar repair operations.

4. The system of claim 3, wherein the one or more similar impacted components or the one or more similar repair operations are identified with fuzzy matching.

5. The system of any one of claims 1 to 4, wherein the expected costs are based in part on expected one or more repair operations for the one or more impacted components.

6. The system of any one of claims 1 to 5, wherein the expected costs are based in part on defects determined in captured images of the object.

7. The system of claim 6, wherein the defects are determined by:processing the captured images of the object using a cage for image alignment, the cage defining segments of the object;aligning the captured images of the object onto the cage to identify segments of the object in the captured images; anddetecting one or more defects in the segments of the object in the captured images.

8. The system of claim 7, wherein the cage is selected from a plurality of cages based on at least one of an object identification number, user selection, and system selection.

9. The system of any one of claims 1 to 8, wherein a visual language extractor is used to process the representation of the invoice.

10. The system of claim 9, wherein the visual language extractor comprises a visual language model.

11. The system of claim 9 or 10, wherein the representation of the invoice are further processed using optical character recognition (OCR) and the system further compares output from the visual language extractor and output from the OCR.

12. The system of any one of claims 1 to 11 , wherein the report identifies anomalies between the costs and the expected costs.

13. The system of any one of claims 1 to 12, wherein transmitting the report comprises automatically submitting an estimate.

14. The system of any one of claims 1 to 13, further comprising a user device with an imager to capture the representation of the invoice.

15. The system of any one of claims 1 to 14, wherein the mapping engine uses Categorization of Component and Operational Costs to map the costs.

16. The system of any one of claims 1 to 15, wherein processing the representation of the invoice to extract costs comprises localizing regions of the representation, wherein regions comprise a main table and a summary table.

17. A method for automatically reviewing an invoice for repairing an object, the method comprising:receiving a representation of the invoice;processing the representation of the invoice to extract costs;mapping the costs onto at least one of one or more impacted components and one or more repair operations;comparing the costs to expected costs based on the at least one of one or more impacted components and the one or more repair operations; andtransmitting or storing a report on the comparison.

18. The method of claim 17, wherein the object comprises a vehicle and the costs comprise materials, parts, and labour costs.

19. The method of claim 17 or 18, wherein the expected costs are based in part on historic data of costs of one or more similar impacted components or one or more similar repair operations.

20. The method of any one of claims 17 to 19, further comprising identifying the one or more similar impacted components or the one or more similar repair operations with fuzzy matching.

21. The method of any one of claims 17 to 20, wherein the expected costs are based in part on expected one or more repair operations for the one or more impacted components.

22. The method of any one of claims 17 to 21 , wherein the expected costs are based in part on defects determined in captured images of the object.

23. The method of claim 22, wherein the defects are determined by:processing the captured images of the object using a cage for image alignment, the cage defining segments of the object;aligning the captured images of the object onto the cage to identify segments of the object in the captured images; anddetecting one or more defects in the segments of the object in the captured images.

24. The method of claim 23, wherein the cage is selected from a plurality of cages based on at least one of an object identification number, user selection, and system selection.

25. The method of any one of claims 17 to 24, wherein a visual language extractor is used to process the representation of the invoice.

26. The method of claim 25, wherein the visual language extractor comprises a visual language model.

27. The method of claim 25 or 26, wherein the representation of the invoice are further processed using optical character recognition (OCR) and the system further compares output from the visual language extractor and output from the OCR.

28. The method of any one of claims 17 to 27, wherein the report identifies anomalies between the costs and the expected costs.

29. The method of any one of claims 17 to 28, wherein transmitting the report comprises automatically submitting an estimate.

30. The method of any one of claims 17 to 29, further comprising capturing the representation of the invoice.

31. The method of any one of claims 17 to 30, wherein mapping the costs uses Categorization of Component and Operational Costs.

32. The system of any one of claims 17 to 31 , wherein processing the representation of the invoice to extract costs comprises localizing regions of the representation, wherein regions comprise a main table and a summary table.

33. A system for automatically reviewing an invoice for repairing an object, the system comprising:a user device including an imager;a server having non-transitory computer readable storage in communication with the user device over a network, the non-transitory computer readable storage storing executable instructions for causing one or more processors to:receive a captured image of the invoice from the user device, the image captured using the imager;process the captured image of the invoice using a visual language extractor comprising a visual language model to extract costs by localizing regions in the captured invoice, wherein the localized regions comprise a main table and a summary table;map the costs onto at least one of one or more impacted components and one or more repair operations using a mapping engine, wherein mapping the costs comprises attaching costs in the summary table to the main table; compare the costs to expected costs based on the at least one of the one or more impacted components and the one or more repair operations; and transmit or store a report on the comparison.

34. The system of claim 33, wherein the executable instructions further cause the one or more processors to:receive captured images of the object from the user device, the images captured using the imager;processing the captured images of the object using a cage for image alignment, the cage defining segments of the object;aligning the captured images of the object onto the cage to identify segments of the object in the captured images;detecting one or more defects in the segments of the object in the captured images; and determining the expected costs based in part on the detected one or more defects.

35. The system of claim 33 or 34, wherein mapping the costs comprises detecting duplicate repair operations and including the detected duplicate repair operations in the report on the comparison.

36. A method of providing a reliability pipeline to machine learning predictions, the method comprising:processing data in redundant steps to predict a result;determining a likelihood estimate that the prediction is wrong;flagging the prediction for review if the likelihood is above a threshold.

37. The method of claim 36, wherein:processing data in redundant steps comprises processing different modes of the data to predict two or more results; andthe likelihood estimate is based on a level of difference between each of the two or more results.

38. The method of claim 37, wherein the different modes of data comprise at least one of: visual data and textual data,a visual language extractor output and a character recognition engine output, and a visual invoice mapping output and a defect detection output.

Images