Serverless Data Pipelines for Scalable Client-Specific Analytics

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing data analytics platforms are limited by their architecture, requiring technical expertise, segregating data products, lacking scalability and automation, and insufficient security and data monitoring capabilities.

Innovation Solution

A serverless data analytics platform that automates deployment and scaling, uses metadata-driven flows, and supports flexible access controls to provide secure, scalable data processing across multiple data products.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If existing data analytics platforms use traditional architecture with physical infrastructure, then data processing capability is provided, but scalability is limited and resource requirements are high

Engineering Contradiction:

Improvedata processing capabilityVSAvoidphysical infrastructure requirements

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The patent replaces physical server farms and computing clusters with a serverless cloud-based architecture. The data analytics platform executes workflows as containerized functions on-demand, eliminating the need for dedicated physical infrastructure while maintaining data processing capabilities. This substitution of mechanical/physical systems with virtualized cloud resources resolves the contradiction between providing data processing capability and reducing infrastructure complexity.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

2Ease of operation

If existing data analytics platforms require multiple years of training for users to attain proficiency, then system control is maintained, but ease of operation deteriorates

Engineering Contradiction:

Improveuser proficiency requirementVSAvoidsystem architecture complexity

Core Design Contradiction:

Ease of operationVSDevice complexity

Solution Approach 1:

The patent introduces a visual workflow builder as an intermediary layer between users and the complex serverless infrastructure. Users can design data analytics workflows through intuitive drag-and-drop interfaces without needing to understand underlying container orchestration, function deployment, or cloud resource management. This intermediary abstraction layer significantly reduces the learning curve while the system manages the complexity of executing these workflows on serverless infrastructure.

Inventive Principle:

Principle #24Intermediary (Mediator)

3Adaptability or versatility

If existing data analytics platforms segregate different data products separately, then data organization is maintained, but adaptability across multiple data products deteriorates

Engineering Contradiction:

Improvecross-data product analysis capabilityVSAvoiddata organization structure

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The patent implements a universal workflow execution engine that can process multiple types of data products through a single unified interface. The serverless architecture allows the same infrastructure to handle diverse data formats and analytics requirements by dynamically instantiating appropriate processing functions. This multi-functional approach enables cross-data product analysis while the system automatically manages the organization and routing of different data types through the unified workflow system.

Inventive Principle:

Principle #6Universality (Multi-functionality)

4Productivity

If existing data analytics platforms lack automation, then manual control is maintained, but productivity in data extraction and loading deteriorates

Engineering Contradiction:

Improvedata extraction and loading efficiencyVSAvoidmanual intervention requirement

Core Design Contradiction:

ProductivityVSExtent of automation

Solution Approach 1:

The patent implements self-service automation through the serverless workflow system. The platform automatically provisions computing resources, manages container lifecycles, handles data orchestration, and performs cleanup operations without manual intervention. Workflow definitions automatically trigger appropriate processing functions, and the system self-manages resource allocation and scaling based on workload demands, dramatically improving data extraction and loading productivity while eliminating the need for manual technical specialist intervention.

Inventive Principle:

Principle #25Self-service

Data Source

PatentUS12530359B2Pipeline systems and methods for use in data analytics platforms

Publication Date: 2026.01.20 FIDELITY INFORMATION SERVICES LLC

AI summary

A data analytics system including an append-only first data store accessible to multiple clients and a second data store is disclosed. The data analytics system can be configurable to, in response to receiving first instructions from a first target system of a first client, the first target system separate from the data analytics system, create a first pipeline between the append-only first data store and the second data store. The first pipeline can be configured according to the first instructions to generate a client-specific data object and store the client-specific data object in the second data store. The data analytics system can be configurable to tear down the first pipeline upon completion of storing the client-specific data object in the second data store.