Kubernetes Pod Dataflow Preview for ETL Consistency

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing ETL processes face inaccuracies and inefficiencies due to the use of separate engines for design time and batch time, leading to mismatches in data transformation previews and lack of customization options for users, which can result in inaccurate data processing and limited user control.

Innovation Solution

Implementing a Kubernetes-based system where both design time and batch time operations are performed within the same kubernetes pod instance, allowing for the use of custom code and minimizing latency by caching design time previews, thus ensuring consistency and user control over the dataflow.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Device complexity

If separate engines are used for design time and batch time operations, then system modularity is improved, but data transformation consistency deteriorates

Engineering Contradiction:

Improvesystem modularityVSAvoiddata transformation consistency

Core Design Contradiction:

Device complexityVSManufacturing precision

Solution Approach 1:

The patent merges design time and batch time operations into a single engine, eliminating the consistency issues caused by separate engines while maintaining system functionality through unified code execution for both preview and production data transformations

Inventive Principle:

Principle #5Merging (Combining)

2Reliability

If previews are executed off-device on user's electronic device, then security is improved, but processing accuracy deteriorates

Engineering Contradiction:

ImprovesecurityVSAvoidpreview accuracy

Core Design Contradiction:

ReliabilityVSManufacturing precision

Solution Approach 1:

The patent introduces an intermediary approach where the preview execution environment is coupled to the data source, allowing secure access to data while enabling accurate transformation previews through the same engine that will execute production transformations, without requiring data to leave the secure environment

Inventive Principle:

Principle #24Intermediary (Mediator)

3Loss of time

If design time previews are cached, then response time is improved, but memory usage increases

Engineering Contradiction:

Improvepreview response timeVSAvoidmemory consumption

Core Design Contradiction:

Loss of timeVSQuantity of substance

Solution Approach 1:

The patent implements caching with configurable parameters including cache size limits, expiration times, and validity conditions, allowing the system to balance between response time improvement and memory consumption by dynamically adjusting cache behavior based on resource availability and usage patterns

Inventive Principle:

Principle #35Parameter changes

Data Source

PatentUS11755608B2Interactive dataflow preview

Publication Date: 2023.09.12 SALESFORCE INC

AI summary

DESCRIBED HEREIN ARE SYSTEMS, APPARATUS, METHODS AND COMPUTER PROGRAM PRODUCTS FOR IMPLEMENTING DESIGN TIME AND BATCH TIME FOR AN EXTRACT, TRANSFORM, LOAD (ETL) PROCESS. WHEN A SESSION IS ESTABLISHED, A KUBERNETES POD INSTANCE MAY BE EXCLUSIVELY ASSOCIATED WITH A USER FOR THE USER'S SESSIONS. DESIGN TIME AND BATCH TIME MAY BOTH BE PERFORMED WITHIN THE KUBERNETES POD INSTANCE. AS SUCH, A SERVICE PROVIDER MAY PROVIDE A SECURE DATAFLOW PREVIEW. FURTHERMORE, THE DATAFLOW MAY BE CACHED. CACHING OF THE DATAFLOW ALLOWS THE SERVICE PROVIDER TO MORE QUICKLY PROVIDE FOLLOW UP PREVIEWS, DECREASING LATENCY.