Check Field Detection Using VLM-MLLM Zero-Shot Bounding Box Selection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional machine learning-based object detection models for check field extraction require fine-tuning on large, annotated datasets, which are difficult to curate due to privacy constraints and diverse visual structures of checks, limiting scalability and availability.
Innovation Solution
A training-free framework combining a visual-language model (VLM) and a multimodal large language model (MLLM) for zero-shot object detection on bank checks, leveraging the VLM to propose bounding boxes and the MLLM to select the correct ones without task-specific training.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional machine learning-based object detection models are used for check field extraction, then detection accuracy can be achieved through fine-tuning, but the requirement for large annotated datasets worsens scalability and availability due to privacy constraints and diverse visual structures
Solution Approach 1:
The patent introduces CLIP (Contrastive Language-Image Pre-training) as an intermediary model that bridges the gap between visual check images and textual field descriptions. CLIP enables zero-shot detection by learning a joint embedding space where images and text prompts can be compared without task-specific fine-tuning, thus achieving adaptability across diverse check formats without requiring annotated datasets for each variant
Solution Approach 2:
The patent performs preliminary action by pre-training the CLIP model on large-scale image-text pairs before deployment. This pre-training establishes a robust foundation for semantic matching that enables the model to generalize to unseen check formats and field types without requiring further fine-tuning on domain-specific annotated data
2Reliability
If traditional object detection models are fine-tuned on annotated datasets, then task-specific performance improves, but the complexity of data curation and model training increases
Solution Approach 1:
The patent enables self-service by allowing the detection system to automatically adapt to new check formats and field types through zero-shot learning. The CLIP model inherently possesses the capability to handle diverse check structures without requiring manual fine-tuning, eliminating the need for complex data curation pipelines and model retraining procedures
Solution Approach 2:
The patent achieves universality by designing a single CLIP-based detection framework that can handle multiple check formats, field types, and document structures simultaneously. The model's ability to process arbitrary text prompts allows it to detect various field types (account numbers, dates, amounts, etc.) across different check designs without requiring separate specialized models
Data Source
AI summary
A first Vision-Language Model (VLM) in a first branch identifies a first set of fields in input data using a visualized first set of bounding boxes (BB). The first VLM labels and outputs a labeled first set of fields. A first agentic AI in the first branch localizes and outputs an identified field as a desired type of filed using a visualized identified BB. A second VLM in a second branch identifies a second set of fields in the input data using a visualized second set of BBs. An MLLM uses the input data with the second set of BBs to output a set of recognizing field from the second set of BBs. A second agentic AI in the second branch and labels a target field. A training data set is formed by combining the input data, labeled identified field, and the labeled target field.