AR Disassembly Support With Image-Based LLM Guidance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing disassembly support methods for product repair lack automation and require significant manual adjustments for new products, relying heavily on prior information like CAD models that are often unavailable, and struggle to generate accurate visualization in augmented reality applications.
Innovation Solution
An automated disassembly support method using a multimodal Large Language Model (LLM) to generate and visualize disassembly steps in augmented reality, enabling adaptive support for new products without manual setup, utilizing image capture, LLM prompts, and visualization modules to guide repair workers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional AR apps with hard-coded disassembly sequences are used, then disassembly guidance can be provided for known products, but adapting to new products is very time-consuming and costly
Solution Approach 1:
The system enables self-service by allowing the multimodal LLM to automatically generate disassembly sequences and AR visualizations for new products without requiring manual programming or configuration. The system serves itself by processing product images and autonomously creating the guidance content, eliminating the need for time-consuming manual adaptation for each new product.
Solution Approach 2:
The system changes the fundamental parameter of how disassembly guidance is created - transitioning from hard-coded sequences to AI-generated sequences based on image input. This parameter change enables the system to adapt to new products dynamically without manual reprogramming, significantly reducing adaptation time while maintaining guidance accuracy.
2Adaptability or versatility
If automatic generation of disassembly sequences is attempted, then adaptation to new products becomes faster, but prior information like CAD models is required which is usually unavailable
Solution Approach 1:
The system extracts only the essential visual information needed for disassembly guidance by capturing images of the actual product. This eliminates the requirement for complex prior information such as CAD models, while still enabling automatic generation of disassembly sequences. The system takes out the unnecessary complexity of requiring structured technical data and works directly with visual input.
Solution Approach 2:
The multimodal LLM acts as an intermediary that bridges the gap between simple product images and complex disassembly sequences. It processes the visual input and translates it into structured disassembly instructions and AR visualizations without requiring the input to be in a complex format, thus reducing information requirements while maintaining adaptability.
3Measurement precision
If detailed text and image repair manuals are used, then disassembly accuracy can be maintained, but following the steps is cumbersome and time-consuming
Solution Approach 1:
The system replaces the mechanical process of manually reading and following text/image instructions with an automated AR-based visual guidance system. The multimodal LLM generates step-by-step disassembly instructions with associated AR visualizations that overlay directly on the product, eliminating the need for workers to manually cross-reference manuals and significantly improving ease of operation while maintaining precision.
Solution Approach 2:
The system transitions from two-dimensional static manuals to three-dimensional interactive AR visualizations. By overlaying disassembly guidance directly onto the physical product in the worker's field of view, the system creates a new dimensional experience that makes following steps intuitive and natural, reducing cognitive load and improving both accuracy and ease of operation.
4Reliability
If AR visualization is generated for each disassembly step, then guidance quality is improved, but processing time and computational resources increase
Solution Approach 1:
The system performs preliminary actions by pre-processing the product image and pre-generating the disassembly sequence before the actual disassembly process begins. The multimodal LLM analyzes the product structure in advance and creates the complete guidance sequence, allowing for efficient real-time visualization during disassembly without compromising productivity.
Solution Approach 2:
The system implements dynamic generation of AR visualizations based on the current disassembly step and product state. Rather than generating all visualizations statically in advance, the system dynamically adapts the visualization content to match the current context, optimizing both quality and processing efficiency by generating only the necessary visual information at each step.
Data Source
Figure 1
AI summary
The invention relates to a disassembly support method comprising the following steps: a) Visual capture of at least a part of a product to be disassembled by an image capture device, in particular integrated into AR glasses; b) Generation of image data for a photograph of the part; c) Prompt for input of a multimodal LLM by an LLM module; d) Transmission of the image data to the LLM module and generation of the prompt at least based on the image data; e) Generation of output containing at least one disassembly step by the multimodal LLM to form disassembly data that at least correlates with the disassembly step based on the prompt; f) Transmission of the data to a visualization module, in particular an AR module; g) Generation of visualization data that correlates at least with the disassembly data by the visualization module; h) Transmission of the visualization data to a visualization module, in particular integrated into the AR glasses.Image generation device, i) generating an image based on the visualization data, j) outputting at least the generated image by the image generation device, k) detecting a trigger signal that correlates at least with the end, l) repeating the preceding steps until a termination criterion is reached. The invention further relates to a disassembly support arrangement for carrying out the method.