Automated Stream Processing Graph Assembly via Semantic Matching

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Stream processing systems face challenges in constructing processing graphs that satisfy user queries due to the large number of disparate data sources and processing elements, requiring manual assembly which is impractical and resource-intensive, especially when dealing with unstructured data and dynamic changes in components.

Innovation Solution

A method and system for assembling stream processing applications using semantic descriptions of data sources, components, and processing requests, where graph patterns in RDF or OWL languages describe output capabilities and input requirements, allowing automated assembly of processing graphs that match user-defined goals and constraints.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If manual assembly of processing graphs is performed, then expertise and control are improved, but time consumption and resource intensity increase significantly

Engineering Contradiction:

ImproveexpertiseVSAvoidtime consumption

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The system enables self-service through automated graph assembly. The processing graph generator automatically constructs processing graphs by matching user queries with available data sources and processing elements, eliminating the need for manual expert intervention while maintaining solution quality through semantic matching algorithms

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

A processing graph generator acts as an intermediary between user queries and the stream processing system. This intermediary component translates high-level user requirements into concrete processing graphs by selecting appropriate components and configuring their interconnections, thereby reducing both time consumption and the need for expert knowledge

Inventive Principle:

Principle #24Intermediary (Mediator)

2Adaptability or versatility

If the number of data sources and processing elements increases, then system capability and versatility are improved, but the complexity of constructing processing graphs increases

Engineering Contradiction:

Improvesystem capabilityVSAvoidgraph construction complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The system segments the complex task of processing graph construction into manageable components: query parsing, component selection, connection establishment, and configuration. The processing graph generator handles each segment systematically, making the overall process tractable even with large numbers of available data sources and processing elements

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The processing graph generator serves as an intermediary that manages the complexity of matching user queries with the large pool of available components. It automatically performs the complex matching and configuration tasks, transforming the complexity from a manual burden into an automated process that scales with system capability

Inventive Principle:

Principle #24Intermediary (Mediator)

3Productivity

If automated assembly is implemented, then time consumption and resource utilization are improved, but the requirement for semantic description capabilities increases

Engineering Contradiction:

Improveassembly speedVSAvoidsemantic description requirement

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

Semantic descriptions of data sources, processing elements, and their capabilities are prepared in advance. This preliminary action creates a structured knowledge base that enables rapid automated assembly without requiring complex semantic processing during runtime, thus improving productivity while managing the complexity of semantic descriptions

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system replaces manual expert judgment (mechanical system) with automated semantic matching algorithms. The processing graph generator uses structured semantic descriptions and algorithmic matching to automatically assemble processing graphs, achieving high productivity while the complexity of semantic descriptions is managed through formalized representation schemes

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Data Source

PatentUS8166465B2Method and system for composing stream processing applications according to a semantic description of a processing goal

Publication Date: 2012.04.24 INTERNATIONAL BUSINESS MACHINE CORPORATION

AI summary

A method for assembling a stream processing application in which data source descriptions, component descriptions and a stream processing request are input and used to assemble a stream processing graph. Each of the data source descriptions includes a graph pattern that semantically describes an output of a data source, each of the component descriptions includes a graph pattern that semantically describes an input of a component and a graph pattern that semantically describes an output of the component, the stream processing request includes a goal that is represented by a graph pattern that semantically describes a desired stream processing outcome and the stream processing graph includes at least one data source or at least one component that satisfies the desired processing outcome.