In-Flight Data Stream Transformations via Serverless FaaS

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing complex event processing (CEP) systems face performance bottlenecks and inefficient memory usage due to the need to consume and store entire data streams before performing transformations, which delays real-time analysis and increases processing power requirements, especially when handling multiple data streams.

Innovation Solution

Implementing a system that uses a virtual private cloud for in-flight transformations of data streams, allowing direct querying and processing of event records in real-time or near real-time through a serverless architecture, with configurable schema and Function as a Service (FaaS) instances to perform necessary transformations and scale resources as needed.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If entire data streams are consumed and stored before transformations, then data completeness is ensured, but processing speed and memory efficiency deteriorate

Engineering Contradiction:

Improvedata completenessVSAvoidprocessing speed

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The system performs preliminary actions by setting up transformation logic and query definitions before data arrives. The transformation framework is pre-configured with schema definitions, transformation rules, and query expressions that will automatically apply to incoming data streams without requiring full data consumption first.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system extracts only the necessary data elements and transformation logic from the complete data stream. Instead of storing entire data streams, it extracts specific event records that match query criteria and applies transformations only to those extracted portions, reducing memory usage while maintaining data completeness for analysis.

Inventive Principle:

Principle #2Taking out (Extraction)

2Reliability

If entire data streams are stored before transformations, then data availability is improved, but memory usage and processing power requirements worsen

Engineering Contradiction:

Improvedata availabilityVSAvoidmemory usage

Core Design Contradiction:

ReliabilityVSQuantity of substance

Solution Approach 1:

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The system performs partial transformations on subsets of data rather than waiting to process complete data streams. By applying transformations to partial data sets that meet query criteria, the system reduces memory requirements while ensuring data availability for immediate analysis when conditions are met.

Inventive Principle:

Principle #16Partial or excessive action

3Manufacturing precision

If transformations are performed after complete stream consumption, then transformation accuracy is ensured, but real-time analysis capability deteriorates

Engineering Contradiction:

Improvetransformation accuracyVSAvoidreal-time analysis delay

Core Design Contradiction:

Manufacturing precisionVSLoss of time

Solution Approach 1:

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system enables continuous transformation and analysis of data streams in real-time or near real-time. By processing data continuously as it arrives rather than waiting for complete streams, the system maintains transformation accuracy through pre-configured rules while eliminating delays associated with batch processing.

Inventive Principle:

Principle #20Continuity of useful action

4Adaptability or versatility

If multiple data streams are processed and stored, then comprehensive analysis capability is improved, but system complexity and resource requirements worsen

Engineering Contradiction:

Improvecomprehensive analysis capabilityVSAvoidsystem complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The system implements a universal transformation framework that can handle multiple data streams through a single unified interface. The configurable schema and transformation logic can be applied across different stream types without requiring separate processing pipelines, reducing system complexity while maintaining comprehensive analysis capability.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Solution Approach 2:

The system segments the processing of multiple data streams into independent, configurable transformation units. Each data stream can be processed through its own schema-defined transformation logic, allowing comprehensive analysis while managing complexity through modular, independent processing segments rather than monolithic system architecture.

Inventive Principle:

Principle #1Segmentation

Data Source

PatentUS12141160B2In-flight transformations over a data stream

Publication Date: 2024.11.12 CAPITAL ONE SERVICES LLC

AI summary

Disclosed herein are system, method, and computer program product embodiments for performing in-flight transformations of data streams. Complex event processing (CEP) operating on event records from various data streams can operate by storing the streamed event record data in a database, and then querying the data to perform a data transformation. However, in order to improve the performance of the data transformations and streamline CEP, a serverless architecture is introduced that can perform data transformations directly on streams, using attributes of the streamed record data defined in a schema. The resulting transformed data can then be provided by the serverless architecture to the CEP for direct access of the data most needed by the CEP.