Multi-Modal Utility Asset Search with Tokenized Visual Indexing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing image databases of utility assets face challenges in efficiently searching for specific defect types and asset classes due to the large volume of images and diverse representation of feature data, making it difficult to determine patterns or insights for improving electric grid operations.
Innovation Solution
A multi-modal search system that builds a token-based search index using machine learning networks to generate image and textual tokens from utility asset images, allowing for efficient retrieval of candidate images based on textual and visual features, and overlays them on a geographical map for dynamic display.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a traditional image database stores all utility asset images with full feature data, then complete information is preserved, but searching becomes inefficient and computationally expensive
Solution Approach 1:
The patent extracts essential visual features from utility asset images and stores them as compact feature vectors in a separate index structure, rather than storing complete image data. This extraction allows the system to perform searches using only the essential features, dramatically reducing computational load while maintaining search effectiveness
Solution Approach 2:
The patent creates simplified copies of image data in the form of feature vectors and embeddings that capture the essential characteristics of utility assets. These copies enable efficient comparison and matching operations without requiring access to the original large-volume image data, thus improving search productivity
2Measurement precision
If the system processes and analyzes all images in real-time, then detection accuracy is maximized, but processing time increases significantly
Solution Approach 1:
The patent performs preliminary processing of utility asset images by pre-computing feature vectors, embeddings, and visual characteristics before actual search or analysis operations. This preliminary action includes generating object detection annotations and extracting key features in advance, so that when queries are executed, the system only needs to perform lightweight comparisons rather than full image analysis
Solution Approach 2:
The patent segments the image analysis process into distinct stages: pre-computation of features, storage in index structures, and query-time matching. This segmentation allows the computationally intensive feature extraction to be performed once during preprocessing, while subsequent searches only require efficient vector comparisons, thereby reducing overall processing time
3Measurement precision
If the system stores detailed annotations and bounding boxes for every image, then object detection precision is improved, but data complexity and storage requirements increase
Solution Approach 1:
The patent transforms detailed image annotations and bounding box data into standardized parameter representations such as feature vectors with fixed dimensions and normalized values. This parameter transformation maintains the precision needed for object detection while organizing data in a uniform structure that simplifies storage, indexing, and retrieval operations
Data Source
AI summary
This disclosure describes systems and methods for multi-modal search-based object detection and electric grid object search. Annotations and bounding boxes for images in an image database are determined. A first subset of images is determined from the images that share annotations. A textual token representing the first subset of images is generated and stored in a search index. A second subset of images that share visual features is determined from image pixels enclosed by the bounding boxes. An image token is generated based on the second subset of images and the shared visual features. A user interface configured to receive a search query input is provided for display on a user device. Search tokens are generated based on the search query input. A candidate image is identified and provided for display within the user interface at a position within a respective region of a geographic map of an electric grid.