Semantic Model Fusion for Multi-Modal Situational Awareness

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing systems struggle to integrate diverse and heterogeneous data sources, such as unstructured free text documents and IoT sensor data, for effective situational awareness and problem diagnosis due to their incompatibility with machine learning and data analytics.

Innovation Solution

A semantic model is created using a situational awareness engine that extracts entities and relationships from multiple data sources through services like table-to-graph, event-to-graph, sensor-to-graph, text-to-graph, and image-to-graph, with human expert input to refine and reconcile the data, forming a knowledge graph for comprehensive situational awareness.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Loss of information

If multiple heterogeneous data sources (IoT sensor data, unstructured text documents, logs, reports) are integrated for situational awareness, then the completeness and value of information is improved, but the complexity of data integration and processing increases

Engineering Contradiction:

Improveinformation completenessVSAvoiddata integration complexity

Core Design Contradiction:

Loss of informationVSDevice complexity

Solution Approach 1:

The patent employs an intermediary layer (data integration service and semantic model) that mediates between heterogeneous data sources and the situational awareness application. This intermediary transforms diverse data formats (IoT sensor data, unstructured text, logs, reports) into a unified semantic representation, hiding the heterogeneity from users while preserving information completeness from all source types

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The system changes the parameter of data representation by transforming raw heterogeneous data into standardized semantic model instances. Each data source is converted from its native format into structured semantic representations with consistent schemas, enabling unified processing while maintaining the original information content

Inventive Principle:

Principle #35Parameter changes

2Measurement precision

If unstructured free text documents (logs, reports, process flow diagrams) are included in data analysis, then the diagnostic capability is improved, but the difficulty of data processing increases

Engineering Contradiction:

Improvediagnostic accuracyVSAvoiddata processing difficulty

Core Design Contradiction:

Measurement precisionVSDifficulty of detecting and measuring

Solution Approach 1:

A text processing intermediary service is introduced that specifically handles unstructured free text documents, logs, reports, and process flow diagrams. This service extracts relevant information and transforms it into structured semantic model instances, making unstructured data amenable to machine learning and data analytics while preserving diagnostic value

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The system replaces manual text analysis with automated natural language processing and text-to-graph services. These computational methods automatically extract entities and relationships from unstructured text, substituting human effort with algorithmic processing that scales to large volumes of documentation

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

3Ease of manufacture

If a standardized terminology and information model are imposed on all team members, then the ease of data integration is improved, but the adaptability to individual preferences is reduced

Engineering Contradiction:

Improvedata integration easeVSAvoidlanguage flexibility

Core Design Contradiction:

Ease of manufactureVSAdaptability or versatility

Solution Approach 1:

The semantic model serves as an intermediary representation layer that enables standardized data integration while preserving individual team members' preferred languages and information models. The system maps diverse source representations to the semantic model without requiring users to adopt standardized terminology in their original data sources

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

Instead of requiring data sources to conform to a standardized model, the system inverts the approach by having the integration layer adapt to diverse source formats and translate them into the unified semantic representation. This reversal maintains source independence while achieving integration goals

Inventive Principle:

Principle #13The other way round (Inversion)

Data Source

PatentUS12406142B2Situational awareness by fusing multi-modal data with semantic model

Publication Date: 2025.09.02 INTERNATIONAL BUSINESS MACHINE CORPORATION

AI summary

A method is provided for creating a semantic model for submitting search queries thereto. The method includes an act of receiving data from one or more input sources in an entity and relationship capture service of a situational awareness engine. The method further includes an act of extracting entities and relationships between the entities in two or more extraction services, where the two or more extraction services include at least two of a table-to-graph service, an event-to-graph service, a sensor-to-graph service, a text-to-graph service, and an image-to-graph service. The method includes an act of generating a semantic model based on fusion and labeling the extracted data provided by the at least two extraction services, where the semantic model can receive a search query and respond to the search query based on the generated semantic model.