Multimodal Spare Part Identification for Ambiguous Visual Matches
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for identifying spare parts in industrial facilities face challenges due to similar appearances and non-intuitive naming conventions, leading to inefficiencies and high costs in determining replacement parts, especially with fluctuating staff and machine learning models requiring extensive training.
Innovation Solution
A multimodal large language model (LLM) processes text, images, and vocal sequences to generate a textual description of a spare part, using k-Nearest Neighbors (kNN) search in a vector database to accurately map the part to a model in a catalogue, leveraging zero-shot learning and integrating text and visual information.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If picture-based identification is used for spare parts, then visual comparison is enabled, but identification becomes ambiguous when parts look alike
Solution Approach 1:
The patent transitions from 2D image-based identification to multi-dimensional identification by incorporating textual descriptions, vocal sequences, and structured part attributes. This dimensional expansion allows the system to distinguish between visually similar parts by comparing them across multiple feature spaces simultaneously, resolving the ambiguity that plagues pure image-matching approaches.
Solution Approach 2:
The patent introduces a multimodal large language model as an intermediary that transforms diverse input modalities (images, text, voice) into a unified textual description. This intermediary layer synthesizes information from multiple sources to create a comprehensive part representation, enabling more accurate identification than any single modality could provide alone.
2Extent of automation
If machine learning models are used for part identification, then automated identification is achieved, but extensive training is required and administration becomes difficult
Solution Approach 1:
The patent replaces traditional machine learning model training mechanisms with a knowledge-based approach using a multimodal large language model. Instead of requiring extensive supervised training with labeled datasets, the system leverages the model's pre-trained language understanding and reasoning capabilities to perform part identification through natural language processing of part descriptions and specifications.
Solution Approach 2:
The system enables users to perform part identification themselves by providing intuitive interfaces where users can input part descriptions, upload images, or record vocal sequences. The multimodal LLM automatically processes these inputs and returns identification results, eliminating the need for users to understand or configure complex machine learning models while maintaining high automation levels.
3Loss of information
If database search by name or machine is provided, then information access is improved, but identification becomes cumbersome when precise name is unknown or multiple parts exist
Solution Approach 1:
The patent creates a universal identification system that accepts multiple input types (text descriptions, images, vocal sequences) and processes them through a single multimodal LLM interface. This multi-functional approach eliminates the need for users to navigate different search methods or know the precise naming conventions of parts, as the system can interpret various input formats and return accurate identification results.
Solution Approach 2:
The system transforms the identification problem by changing the input parameters from requiring precise part names or codes to accepting natural language descriptions, images, or voice inputs. The multimodal LLM processes these varied parameters and converts them into accurate part identifications, significantly reducing the time users spend browsing databases while maintaining comprehensive information access.
Data Source
Figure 1
Figure 2~4
AI summary
The invention relates to a computer-implemented method, a computer-implemented device, a system and a computer program product for mapping a real-world part to a model in a catalogue comprising receiving, by a multimodal large language model, LLM, a query for mapping the part to the model, wherein the request comprises a text and/or a voice sequence and/or an image associated with the part, transforming, by the multimodal LLM, the received query to a textual description of the part, identifying, based on the textual description, a model in the catalogue onto which the part may be mapped, wherein the identifying comprises comparing the textual description with a respective textual description of the models in the catalogue; providing information associated with the identified model to a user of the computer-implemented method