Stream Processing Resource Version Consistency

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Stream processing systems face compatibility issues due to resource type changes, leading to incompatibility and communication failures between processing elements, as new and older resource type definitions are mixed, causing disruptions in job execution.

Innovation Solution

A method is implemented to maintain version consistency of resource types by using a tag mapper to determine the appropriate resource type definition based on a timestamp associated with each processing element, ensuring that all elements within a job use compatible resource versions, and requesting resources from a resource manager to align with these versions.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If stream processing jobs use updated resource type definitions to benefit from new features and improvements, then resource functionality and capabilities are enhanced, but compatibility issues arise between processing elements using different resource versions, leading to communication failures and job execution disruptions

Engineering Contradiction:

Improveresource functionalityVSAvoidjob execution stability

Core Design Contradiction:

Adaptability or versatilityVSReliability

Solution Approach 1:

The system performs preliminary version assignment by tagging each processing element with a resource version tag at job submission time, before actual resource allocation occurs. This preliminary action ensures that all processing elements in a job are pre-coordinated to use the same resource version, preventing compatibility issues before they can manifest during execution.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

A resource version tag acts as an intermediary between the job submission system and the actual resource allocation. This intermediary mechanism translates job requirements into specific versioned resource requests, ensuring that resource type changes are isolated from applications while maintaining controlled access to updated resource definitions through explicit version specification.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Reliability

If the system maintains strict version consistency across all processing elements in a job, then compatibility and communication reliability are improved, but the system loses flexibility to utilize newly updated resource types that could provide performance improvements

Engineering Contradiction:

Improvecommunication reliabilityVSAvoidjob processing efficiency

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The system implements dynamic version selection where the resource version is not fixed at compile time but is determined at job submission time based on current system state and requirements. This allows the system to adaptively choose between maintaining version consistency for reliability or utilizing newer resource versions for improved performance, depending on the specific job requirements and available resources.

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The system changes the version parameter of resource types based on job requirements and system state. By making the version parameter variable rather than static, the system can optimize for either reliability or productivity depending on the situation, allowing processing elements to use newer resource versions when performance is prioritized while maintaining consistency when reliability is the primary concern.

Inventive Principle:

Principle #35Parameter changes

3Adaptability or versatility

If processing elements are allowed to use different resource type definitions, then the system can leverage latest resource improvements and maintain operational flexibility, but incompatibility issues arise causing communication failures between processing elements within the same job

Engineering Contradiction:

Improveoperational flexibilityVSAvoidcommunication failures

Core Design Contradiction:

Adaptability or versatilityVSObject-affected harmful factors

Solution Approach 1:

The system segments the resource type space into distinct versioned categories, where each processing element is assigned to a specific resource version segment. This segmentation prevents incompatible resource types from being mixed within the same job by creating clear boundaries between different version groups, while still allowing the system to leverage multiple resource versions across different jobs for operational flexibility.

Inventive Principle:

Principle #1Segmentation

Data Source

PatentUS11650858B2Maintaining stream processing resource type versions in stream processing

Publication Date: 2023.05.16 INTERNATIONAL BUSINESS MACHINE CORPORATION

AI summary

A method for maintaining version consistency of resources. The method provides for one or more processors to receive a submitted request to run a job in which the job includes a processing element and a timestamp associated with running the job. Identification of a resource type associated with the processing element is determined, based on a tag included in the job, associated with the processing element. A version of the resource type of the processing element is determined, based on a mapping of the tag associated with the identified resource type and the timestamp of the job. The resource type of the determined version is requested from a resource manager, and responsive to a confirmation of assigning the version of the resource type from the resource manager, the process element of the job is performed on the version of the resource type assigned by the resource manager.