Stream Message Linearization With File Staging and Normalization

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing stream-processing systems face challenges in handling diverse data formats and protocols from external data sources, leading to inefficiencies in processing large volumes and varying velocities of messages.

Innovation Solution

A distributed stream-processing system that categorizes tasks into specific categories for data processing, including in-protocol, unbundling, unpacking, message normalization, and validation, allowing for efficient handling of various data formats and protocols through a sequence of tasks and a file staging area for handling unprocessable data objects.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If stream-processing systems handle diverse data formats and protocols from external data sources, then adaptability is improved, but processing efficiency deteriorates due to the need to accommodate multiple formats and protocols

Engineering Contradiction:

Improveability to handle diverse data formats and protocolsVSAvoidprocessing efficiency

Core Design Contradiction:

Adaptability or versatilityVSProductivity

Solution Approach 1:

The processing pipeline is segmented into distinct stages: data reception, format detection, normalization, validation, and output. Each stage handles a specific aspect of data processing, allowing the system to efficiently handle diverse formats through specialized processing steps rather than monolithic processing

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

A standardized internal data format acts as an intermediary between diverse external data sources and the processing tasks. External data is received in various formats, normalized to a common internal representation, then passed to tasks that operate on this unified format, thereby improving processing efficiency while maintaining adaptability

Inventive Principle:

Principle #24Intermediary (Mediator)

2Ease of operation

If tasks are bound to specific data sources with fixed formats, then processing simplicity is improved, but system flexibility deteriorates when data sources change formats or protocols

Engineering Contradiction:

Improveprocessing simplicityVSAvoidflexibility when data sources change formats

Core Design Contradiction:

Ease of operationVSAdaptability or versatility

Solution Approach 1:

The data reception component is designed with multi-functionality to handle multiple data formats and protocols through a unified interface. Format detection and normalization mechanisms enable the same reception component to accommodate various data sources without requiring separate processing logic for each format, thereby maintaining simplicity while improving flexibility

Inventive Principle:

Principle #6Universality (Multi-functionality)

Solution Approach 2:

The system dynamically adjusts processing parameters based on the detected data format. When data sources change formats, the system detects the new format and modifies normalization and validation parameters accordingly, allowing flexible adaptation without changing the fundamental processing architecture or task bindings

Inventive Principle:

Principle #35Parameter changes

3Loss of information

If messages are processed in their original formats from various data sources, then data fidelity is preserved, but task compatibility issues arise when tasks cannot handle all formats

Engineering Contradiction:

Improvedata fidelityVSAvoidtask compatibility

Core Design Contradiction:

Loss of informationVSReliability

Solution Approach 1:

Format normalization is performed as a preliminary action before data reaches processing tasks. The system detects the original data format, normalizes it to a standardized internal representation, and then passes it to tasks. This preliminary normalization ensures task compatibility while preserving data fidelity through accurate format detection and faithful conversion to the standardized format

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system incorporates feedback mechanisms where format detection results feed back into the normalization process. When tasks encounter format incompatibility, the system can detect this and adjust the normalization parameters or data transformation logic, ensuring reliable processing while maintaining data accuracy through iterative refinement

Inventive Principle:

Principle #23Feedback

Data Source

PatentUS12353929B2System and method for linearizing messages from data sources for optimized high-performance processing in a stream processing system

Publication Date: 2025.07.08 FAIR ISAAC & CO INC

AI summary

A data object from a data source is received by a distributed process in a data stream. The distributed process has a sequence of categories, each category containing one or more tasks that operate on the data object. The data object includes files that can be processed by the tasks. If the task is able to operate on the data object, then the data object is passed to the task. If the task is unable to operate on the data object, then the files in the data object are passed to a file staging area of the distributed process and stored in memory. The files in the file staging area are passed, in sequence, from the file staging area to the task that was unable to operate on the data object. The data object is outputted to a next category or data sink after being operated on by the task.