Multimodal Content Comparison Using Ranked Common Concepts
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning workflows face challenges in interpreting and addressing differences between synthetic and real data, or differences in model performance on correctly versus incorrectly predicted data, particularly in understanding domain shifts during deployment, without effective natural language descriptions.
Innovation Solution
Utilizing multi-modal foundation models to process data modalities like digital images and audio signals, generating element-wise descriptions with data-to-text models, determining common concepts using large language models and text-data-similarity metrics, and ranking these concepts based on average text-data similarities to identify anomalies.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If multi-modal foundation models and large language models are used to generate natural language descriptions of dataset differences, then interpretability and actionability of data analysis is improved, but device complexity and computational resources required increase
Solution Approach 1:
The system segments the complex task of dataset comparison into multiple stages: generating element-wise descriptions with data-to-text models, identifying common concepts with large language models, calculating text-data-similarities, and ranking concepts. This segmentation allows each component to specialize in a specific subtask, improving overall interpretability while managing system complexity through modular architecture.
Solution Approach 2:
The patent introduces intermediate representations including element-wise descriptions, common concepts, and text-data-similarity scores as mediators between the raw datasets and the final natural language explanations. These intermediaries bridge the gap between complex multi-modal data and interpretable outputs, reducing information loss while allowing the system to process data in manageable steps.
2Measurement precision
If element-wise descriptions and common concept analysis are generated for all elements in datasets, then measurement precision of dataset differences is improved, but processing time and computational resources increase
Solution Approach 1:
The system applies partial action by focusing computational resources on identifying and analyzing only the most relevant common concepts that actually differ between datasets. Rather than exhaustively processing all possible concepts for all elements, the method ranks concepts by their average text-data-similarity and selects only those with significant differences, achieving high measurement precision while reducing processing time by avoiding unnecessary computations on irrelevant concepts.
3Reliability
If text-data-similarity metrics are calculated for all common concepts across all dataset elements, then reliability of anomaly detection is improved, but productivity of the processing system decreases
Solution Approach 1:
The system applies local quality by calculating text-data-similarity metrics selectively for specific common concepts that are relevant to particular dataset elements, rather than uniformly for all concepts across all elements. The ranking mechanism identifies local areas of high similarity (or low similarity indicating anomalies) and focuses detailed analysis there, improving detection reliability while maintaining productivity by avoiding exhaustive computation everywhere.
Data Source
Figure 1~2
Figure 3
AI summary
A device, a datastructure, and a computer implemented method for digital content processing, wherein the method comprises providing (202) a first dataset, wherein the first dataset comprises elements, providing (204) a second dataset, wherein the second dataset comprises elements, wherein a digital content of a respective element of the elements comprises a digital image, for example a video image, a radar image, a LiDAR image, an ultrasonic image, a motion image, or a thermal image, or wherein a content of a respective element of the elements comprise a digital audio signal, generating (206), in particular with a data-to-text model, a first set of descriptions, wherein the first set comprises an element-wise description of the elements of the first dataset, wherein the description of the respective element of the first dataset is determined depending on the content of the respective element of the first dataset, generating (208), in particular with the data-to-text model, a second set of descriptions, wherein the second set comprises an element-wise description of the elements of the second dataset, wherein the description of the respective element of the second dataset is determined depending on the content of the respective element of the second dataset, determining (210), in particular with a large language model, common concepts in the first dataset that are non-existent in the second dataset or less frequent in the second dataset than in the first dataset, determining (212), in particular with a text-data-similarity metric, for the elements of the first dataset a first plurality of text-data-similarities, wherein the first plurality comprises the element-wise and common concept-wise text-data-similarity of pairs of the content of one element of the first dataset and one common concept, determining (214), in particular with the text-data-similarity metric, for the elements of the second dataset a second plurality of text-data-similarities, wherein the second plurality comprises the element-wise and common concept-wise text-data-similarity of pairs of the content of one element of the second dataset and one common concept, determining (216) for the first plurality common concept-wise the average text-data similarity that is associated with the respective common concept according to the first plurality, determining (218) for the second plurality common concept-wise the average text-data similarity that is associated with the respective common concept according to the second plurality, associating (220) the common concepts common concept-wise with a rank, wherein the rank is determined by the average text-data similarities associated with the common concepts according to the first plurality and by the average text-data similarities associated with the common concepts according to the second plurality, selecting (222) at least one common concept depending on the ranks that are associated with the common concepts, and outputting (224) the selected at least one common concept.